Oct 30 00:40:47.190367 ip-10-0-134-210 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 00:40:47.597988 ip-10-0-134-210 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:40:47.597988 ip-10-0-134-210 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 00:40:47.597988 ip-10-0-134-210 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:40:47.597988 ip-10-0-134-210 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:40:47.601645 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601551    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:40:47.601645 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601640    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:40:47.601645 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601647    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:40:47.601645 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601651    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:40:47.601910 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601656    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:40:47.601910 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601660    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:40:47.601910 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601666    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:40:47.601910 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601670    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:40:47.601910 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601680    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:40:47.601910 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601684    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:40:47.601910 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601688    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:40:47.601910 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601692    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:40:47.601910 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601696    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:40:47.601910 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601700    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:40:47.601910 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601704    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:40:47.601910 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601708    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:40:47.601910 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601712    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:40:47.601910 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601716    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:40:47.601910 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601720    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:40:47.601910 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601730    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:40:47.601910 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601734    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:40:47.601910 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601738    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:40:47.601910 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601742    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:40:47.601910 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601747    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:40:47.602776 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601751    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:40:47.602776 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601755    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:40:47.602776 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601759    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:40:47.602776 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601762    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:40:47.602776 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601766    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:40:47.602776 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601771    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:40:47.602776 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601775    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:40:47.602776 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601779    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:40:47.602776 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601784    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:40:47.602776 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601789    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:40:47.602776 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601793    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:40:47.602776 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601797    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:40:47.602776 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601801    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:40:47.602776 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601806    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:40:47.602776 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601810    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:40:47.602776 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601815    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:40:47.602776 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601819    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:40:47.602776 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601824    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:40:47.602776 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601828    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:40:47.602776 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601833    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:40:47.603741 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601838    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:40:47.603741 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601842    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:40:47.603741 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601846    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:40:47.603741 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601850    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:40:47.603741 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601855    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:40:47.603741 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601859    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:40:47.603741 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601863    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:40:47.603741 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601868    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:40:47.603741 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601872    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:40:47.603741 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601876    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:40:47.603741 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601881    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:40:47.603741 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601885    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:40:47.603741 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601889    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:40:47.603741 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601894    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:40:47.603741 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601898    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:40:47.603741 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601902    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:40:47.603741 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601906    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:40:47.603741 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601911    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:40:47.603741 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601915    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:40:47.603741 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601919    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:40:47.604630 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601925    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:40:47.604630 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601929    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:40:47.604630 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601933    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:40:47.604630 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601937    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:40:47.604630 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601941    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:40:47.604630 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601946    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:40:47.604630 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601950    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:40:47.604630 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601957    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:40:47.604630 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601965    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:40:47.604630 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601971    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:40:47.604630 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601976    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:40:47.604630 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601981    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:40:47.604630 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601986    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:40:47.604630 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601990    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:40:47.604630 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601995    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:40:47.604630 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.601999    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:40:47.604630 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602003    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:40:47.604630 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602007    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:40:47.604630 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602011    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:40:47.605443 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602016    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:40:47.605443 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602021    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:40:47.605443 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602025    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:40:47.605443 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602029    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:40:47.605443 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602034    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:40:47.605443 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602038    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:40:47.605443 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602043    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:40:47.605443 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602047    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:40:47.605443 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602051    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:40:47.605443 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602056    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:40:47.605443 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602063    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:40:47.605443 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602084    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:40:47.605443 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602089    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:40:47.605443 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602096    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:40:47.605443 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602100    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:40:47.605443 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602105    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:40:47.605443 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602109    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:40:47.605443 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602113    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:40:47.605443 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602117    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:40:47.605916 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602121    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:40:47.605916 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602125    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:40:47.605916 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602130    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:40:47.605916 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.602134    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:40:47.605916 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603434    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:40:47.605916 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603445    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:40:47.605916 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603449    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:40:47.605916 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603454    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:40:47.605916 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603458    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:40:47.605916 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603463    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:40:47.605916 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603467    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:40:47.605916 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603471    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:40:47.605916 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603476    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:40:47.605916 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603480    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:40:47.605916 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603484    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:40:47.605916 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603489    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:40:47.605916 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603494    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:40:47.605916 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603498    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:40:47.605916 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603503    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:40:47.606449 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603507    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:40:47.606449 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603511    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:40:47.606449 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603516    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:40:47.606449 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603520    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:40:47.606449 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603525    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:40:47.606449 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603530    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:40:47.606449 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603534    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:40:47.606449 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603538    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:40:47.606449 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603542    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:40:47.606449 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603547    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:40:47.606449 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603552    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:40:47.606449 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603556    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:40:47.606449 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603560    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:40:47.606449 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603565    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:40:47.606449 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603569    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:40:47.606449 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603573    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:40:47.606449 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603578    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:40:47.606449 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603582    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:40:47.606449 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603588    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:40:47.607274 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603596    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:40:47.607274 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603602    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:40:47.607274 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603608    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:40:47.607274 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603613    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:40:47.607274 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603618    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:40:47.607274 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603623    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:40:47.607274 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603627    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:40:47.607274 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603632    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:40:47.607274 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603636    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:40:47.607274 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603643    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:40:47.607274 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603650    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:40:47.607274 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603655    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:40:47.607274 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603659    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:40:47.607274 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603664    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:40:47.607274 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603668    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:40:47.607274 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603673    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:40:47.607274 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603677    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:40:47.607274 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603682    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:40:47.607274 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603687    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:40:47.608042 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603691    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:40:47.608042 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603695    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:40:47.608042 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603700    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:40:47.608042 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603704    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:40:47.608042 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603709    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:40:47.608042 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603714    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:40:47.608042 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603718    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:40:47.608042 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603722    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:40:47.608042 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603726    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:40:47.608042 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603730    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:40:47.608042 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603734    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:40:47.608042 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603738    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:40:47.608042 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603742    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:40:47.608042 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603746    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:40:47.608042 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603750    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:40:47.608042 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603754    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:40:47.608042 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603758    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:40:47.608042 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603762    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:40:47.608042 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603766    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:40:47.608042 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603770    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:40:47.608576 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603774    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:40:47.608576 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603779    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:40:47.608576 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603783    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:40:47.608576 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603788    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:40:47.608576 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603792    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:40:47.608576 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603796    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:40:47.608576 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603801    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:40:47.608576 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603805    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:40:47.608576 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603809    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:40:47.608576 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603814    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:40:47.608576 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603818    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:40:47.608576 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603822    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:40:47.608576 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603827    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:40:47.608576 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603831    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:40:47.608576 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603836    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:40:47.608576 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603840    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:40:47.608576 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603845    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:40:47.608576 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603849    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:40:47.608576 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603854    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:40:47.608576 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603858    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:40:47.609308 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603862    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:40:47.609308 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603866    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:40:47.609308 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603871    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:40:47.609308 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603875    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:40:47.609308 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603879    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:40:47.609308 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603883    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:40:47.609308 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603888    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:40:47.609308 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603892    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:40:47.609308 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603897    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:40:47.609308 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603901    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:40:47.609308 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603907    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:40:47.609308 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603911    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:40:47.609308 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.603915    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:40:47.609308 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604809    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:40:47.609308 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604820    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:40:47.609308 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604825    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:40:47.609308 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604829    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:40:47.609308 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604834    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:40:47.609308 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604838    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:40:47.609308 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604843    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:40:47.609877 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604847    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:40:47.609877 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604852    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:40:47.609877 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604856    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:40:47.609877 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604861    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:40:47.609877 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604865    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:40:47.609877 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604870    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:40:47.609877 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604875    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:40:47.609877 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604879    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:40:47.609877 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604883    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:40:47.609877 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604887    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:40:47.609877 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604891    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:40:47.609877 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604896    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:40:47.609877 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604900    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:40:47.609877 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604905    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:40:47.609877 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604909    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:40:47.609877 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604913    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:40:47.609877 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604918    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:40:47.609877 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604922    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:40:47.609877 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604927    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:40:47.610368 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604931    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:40:47.610368 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604936    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:40:47.610368 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604940    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:40:47.610368 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604945    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:40:47.610368 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604949    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:40:47.610368 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604953    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:40:47.610368 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604958    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:40:47.610368 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604962    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:40:47.610368 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604967    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:40:47.610368 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604972    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:40:47.610368 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604976    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:40:47.610368 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604980    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:40:47.610368 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604985    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:40:47.610368 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604989    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:40:47.610368 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604993    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:40:47.610368 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.604997    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:40:47.610368 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605001    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:40:47.610368 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605007    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:40:47.610368 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605012    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:40:47.610799 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605016    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:40:47.610799 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605020    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:40:47.610799 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605025    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:40:47.610799 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605029    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:40:47.610799 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605033    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:40:47.610799 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605038    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:40:47.610799 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605042    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:40:47.610799 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605046    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:40:47.610799 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605050    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:40:47.610799 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605054    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:40:47.610799 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605058    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:40:47.610799 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605062    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:40:47.610799 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605066    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:40:47.610799 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605087    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:40:47.610799 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605092    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:40:47.610799 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605096    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:40:47.610799 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605101    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:40:47.610799 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605105    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:40:47.610799 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605109    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:40:47.610799 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605114    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:40:47.611278 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605119    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:40:47.611278 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605123    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:40:47.611278 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605127    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:40:47.611278 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605131    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:40:47.611278 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605135    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:40:47.611278 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605142    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:40:47.611278 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605149    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:40:47.611278 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605157    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:40:47.611278 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605163    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:40:47.611278 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605168    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:40:47.611278 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605172    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:40:47.611278 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605176    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:40:47.611278 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605181    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:40:47.611278 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605185    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:40:47.611278 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605189    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:40:47.611278 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605193    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:40:47.611278 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605197    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:40:47.611278 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605201    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:40:47.611278 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605205    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:40:47.611734 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605210    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:40:47.611734 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605214    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:40:47.611734 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605218    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:40:47.611734 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605222    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:40:47.611734 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605229    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:40:47.611734 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605233    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:40:47.611734 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605238    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:40:47.611734 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605242    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:40:47.611734 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605248    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:40:47.611734 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605255    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:40:47.611734 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605260    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:40:47.611734 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605264    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:40:47.611734 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605268    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:40:47.611734 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605273    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:40:47.611734 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605277    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:40:47.611734 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605281    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:40:47.611734 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605285    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:40:47.611734 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605289    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:40:47.611734 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605293    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:40:47.611734 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605297    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:40:47.612213 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605302    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:40:47.612213 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.605306    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:40:47.612213 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606353    2582 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 00:40:47.612213 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606369    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 00:40:47.612213 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606381    2582 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 00:40:47.612213 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606389    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 00:40:47.612213 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606396    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 00:40:47.612213 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606401    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 00:40:47.612213 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606408    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 00:40:47.612213 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606415    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 00:40:47.612213 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606420    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 00:40:47.612213 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606426    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 00:40:47.612213 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606432    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 00:40:47.612213 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606437    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 00:40:47.612213 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606442    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 00:40:47.612213 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606447    2582 flags.go:64] FLAG: --cgroup-root=""
Oct 30 00:40:47.612213 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606454    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 00:40:47.612213 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606458    2582 flags.go:64] FLAG: --client-ca-file=""
Oct 30 00:40:47.612213 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606463    2582 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 00:40:47.612213 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606468    2582 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 00:40:47.612213 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606476    2582 flags.go:64] FLAG: --cluster-domain=""
Oct 30 00:40:47.612213 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606481    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 00:40:47.612213 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606487    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 00:40:47.612738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606492    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 00:40:47.612738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606499    2582 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 00:40:47.612738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606506    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 00:40:47.612738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606511    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 00:40:47.612738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606516    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 00:40:47.612738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606522    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 00:40:47.612738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606527    2582 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 00:40:47.612738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606532    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 00:40:47.612738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606537    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 00:40:47.612738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606542    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 00:40:47.612738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606547    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 00:40:47.612738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606554    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 00:40:47.612738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606559    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 00:40:47.612738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606564    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 00:40:47.612738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606569    2582 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 00:40:47.612738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606574    2582 flags.go:64] FLAG: --enable-server="true"
Oct 30 00:40:47.612738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606579    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 00:40:47.612738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606587    2582 flags.go:64] FLAG: --event-burst="100"
Oct 30 00:40:47.612738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606592    2582 flags.go:64] FLAG: --event-qps="50"
Oct 30 00:40:47.612738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606596    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 00:40:47.612738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606601    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 00:40:47.612738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606606    2582 flags.go:64] FLAG: --eviction-hard=""
Oct 30 00:40:47.612738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606613    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 00:40:47.612738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606618    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 00:40:47.613394 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606623    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 00:40:47.613394 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606629    2582 flags.go:64] FLAG: --eviction-soft=""
Oct 30 00:40:47.613394 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606635    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 00:40:47.613394 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606640    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 00:40:47.613394 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606645    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 00:40:47.613394 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606653    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 00:40:47.613394 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606659    2582 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 00:40:47.613394 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606664    2582 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 00:40:47.613394 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606669    2582 flags.go:64] FLAG: --feature-gates=""
Oct 30 00:40:47.613394 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606676    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 00:40:47.613394 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606681    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 00:40:47.613394 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606686    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 00:40:47.613394 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606692    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 00:40:47.613394 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606697    2582 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 00:40:47.613394 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606702    2582 flags.go:64] FLAG: --help="false"
Oct 30 00:40:47.613394 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606706    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-134-210.ec2.internal"
Oct 30 00:40:47.613394 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606711    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 00:40:47.613394 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606716    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 00:40:47.613394 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606722    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 00:40:47.613394 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606727    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 00:40:47.613394 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606732    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 00:40:47.613394 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606737    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 00:40:47.613394 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606742    2582 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606747    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606751    2582 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606756    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606761    2582 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606766    2582 flags.go:64] FLAG: --kube-reserved=""
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606771    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606776    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606781    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606785    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606791    2582 flags.go:64] FLAG: --lock-file=""
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606797    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606802    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606807    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606815    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606822    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606827    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606832    2582 flags.go:64] FLAG: --logging-format="text"
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606838    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606843    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606848    2582 flags.go:64] FLAG: --manifest-url=""
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606853    2582 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606860    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606866    2582 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606872    2582 flags.go:64] FLAG: --max-pods="110"
Oct 30 00:40:47.613975 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606877    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606882    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606886    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606891    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606896    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606926    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606933    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606950    2582 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606956    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606961    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606966    2582 flags.go:64] FLAG: --pod-cidr=""
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606971    2582 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606976    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606981    2582 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606986    2582 flags.go:64] FLAG: --port="10250"
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606991    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.606995    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07159f16fd508fd11"
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607001    2582 flags.go:64] FLAG: --qos-reserved=""
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607008    2582 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607013    2582 flags.go:64] FLAG: --register-node="true"
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607018    2582 flags.go:64] FLAG: --register-with-taints=""
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607024    2582 flags.go:64] FLAG: --registry-burst="10"
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607031    2582 flags.go:64] FLAG: --registry-qps="5"
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607035    2582 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607040    2582 flags.go:64] FLAG: --reserved-memory=""
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607047    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 00:40:47.614578 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607053    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607058    2582 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607063    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607082    2582 flags.go:64] FLAG: --runonce="false"
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607088    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607093    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607098    2582 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607103    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607108    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607113    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607118    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607123    2582 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607128    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607133    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607138    2582 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607143    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607148    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607152    2582 flags.go:64] FLAG: --system-cgroups=""
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607157    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607166    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607171    2582 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607176    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607182    2582 flags.go:64] FLAG: --tls-min-version=""
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607187    2582 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607194    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 00:40:47.615215 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607198    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 00:40:47.615792 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607203    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 00:40:47.615792 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607208    2582 flags.go:64] FLAG: --v="2"
Oct 30 00:40:47.615792 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607216    2582 flags.go:64] FLAG: --version="false"
Oct 30 00:40:47.615792 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607223    2582 flags.go:64] FLAG: --vmodule=""
Oct 30 00:40:47.615792 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607230    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 00:40:47.615792 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.607236    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 00:40:47.615792 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607468    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:40:47.615792 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607477    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:40:47.615792 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607483    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:40:47.615792 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607488    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:40:47.615792 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607493    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:40:47.615792 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607497    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:40:47.615792 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607501    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:40:47.615792 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607505    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:40:47.615792 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607510    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:40:47.615792 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607514    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:40:47.615792 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607519    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:40:47.615792 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607525    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:40:47.615792 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607533    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:40:47.615792 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607538    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:40:47.615792 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607542    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:40:47.616345 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607546    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:40:47.616345 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607551    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:40:47.616345 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607555    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:40:47.616345 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607559    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:40:47.616345 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607564    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:40:47.616345 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607568    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:40:47.616345 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607572    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:40:47.616345 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607577    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:40:47.616345 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607581    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:40:47.616345 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607587    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:40:47.616345 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607592    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:40:47.616345 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607597    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:40:47.616345 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607601    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:40:47.616345 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607606    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:40:47.616345 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607611    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:40:47.616345 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607615    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:40:47.616345 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607619    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:40:47.616345 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607625    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:40:47.616345 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607630    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:40:47.616345 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607634    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:40:47.616829 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607639    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:40:47.616829 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607643    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:40:47.616829 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607647    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:40:47.616829 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607652    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:40:47.616829 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607656    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:40:47.616829 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607660    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:40:47.616829 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607664    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:40:47.616829 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607668    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:40:47.616829 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607673    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:40:47.616829 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607677    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:40:47.616829 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607681    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:40:47.616829 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607685    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:40:47.616829 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607689    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:40:47.616829 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607694    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:40:47.616829 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607699    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:40:47.616829 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607703    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:40:47.616829 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607707    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:40:47.616829 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607711    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:40:47.616829 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607717    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:40:47.616829 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607722    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:40:47.617325 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607727    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:40:47.617325 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607733    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:40:47.617325 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607737    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:40:47.617325 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607741    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:40:47.617325 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607745    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:40:47.617325 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607750    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:40:47.617325 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607754    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:40:47.617325 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607758    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:40:47.617325 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607762    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:40:47.617325 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607767    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:40:47.617325 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607772    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:40:47.617325 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607776    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:40:47.617325 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607781    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:40:47.617325 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607785    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:40:47.617325 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607790    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:40:47.617325 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607794    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:40:47.617325 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607798    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:40:47.617325 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607802    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:40:47.617325 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607809    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:40:47.617818 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607816    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:40:47.617818 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607822    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:40:47.617818 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607828    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:40:47.617818 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607832    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:40:47.617818 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607836    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:40:47.617818 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607841    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:40:47.617818 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607845    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:40:47.617818 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607850    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:40:47.617818 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607854    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:40:47.617818 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607858    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:40:47.617818 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607863    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:40:47.617818 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607867    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:40:47.617818 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607872    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:40:47.617818 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607876    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:40:47.617818 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607882    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:40:47.617818 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607887    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:40:47.617818 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607891    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:40:47.617818 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607895    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:40:47.617818 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607900    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:40:47.617818 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607904    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:40:47.618319 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607908    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:40:47.618319 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607913    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:40:47.618319 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607917    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:40:47.618319 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607921    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:40:47.618319 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607925    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:40:47.618319 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607929    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:40:47.618319 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607934    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:40:47.618319 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607939    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:40:47.618319 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607943    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:40:47.618319 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607967    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:40:47.618319 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607986    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:40:47.618319 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.607990    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:40:47.618319 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.608035    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:40:47.618319 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608095    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:40:47.618319 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608101    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:40:47.618319 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608105    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:40:47.618696 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608108    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:40:47.618696 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608112    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:40:47.618696 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608115    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:40:47.618696 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608118    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:40:47.618696 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608122    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:40:47.618696 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608127    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:40:47.618696 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608131    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:40:47.618696 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608134    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:40:47.618696 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608137    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:40:47.618696 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608140    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:40:47.618696 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608144    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:40:47.618696 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608147    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:40:47.618696 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608150    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:40:47.618696 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608153    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:40:47.618696 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608156    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:40:47.618696 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608159    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:40:47.618696 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608163    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:40:47.618696 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608165    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:40:47.618696 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608168    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:40:47.619156 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608171    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:40:47.619156 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608174    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:40:47.619156 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608177    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:40:47.619156 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608180    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:40:47.619156 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608183    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:40:47.619156 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608185    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:40:47.619156 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608188    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:40:47.619156 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608191    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:40:47.619156 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608194    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:40:47.619156 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608197    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:40:47.619156 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608200    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:40:47.619156 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608203    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:40:47.619156 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608205    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:40:47.619156 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608208    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:40:47.619156 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608211    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:40:47.619156 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608213    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:40:47.619156 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608216    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:40:47.619156 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608218    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:40:47.619156 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608221    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:40:47.619156 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608224    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:40:47.619635 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608226    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:40:47.619635 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608229    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:40:47.619635 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608232    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:40:47.619635 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608235    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:40:47.619635 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608238    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:40:47.619635 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608240    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:40:47.619635 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608243    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:40:47.619635 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608246    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:40:47.619635 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608249    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:40:47.619635 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608251    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:40:47.619635 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608254    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:40:47.619635 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608256    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:40:47.619635 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608259    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:40:47.619635 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608262    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:40:47.619635 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608264    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:40:47.619635 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608267    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:40:47.619635 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608270    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:40:47.619635 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608273    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:40:47.619635 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608277    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:40:47.619635 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608280    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:40:47.620142 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608283    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:40:47.620142 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608286    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:40:47.620142 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608288    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:40:47.620142 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608291    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:40:47.620142 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608293    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:40:47.620142 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608296    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:40:47.620142 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608299    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:40:47.620142 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608303    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:40:47.620142 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608307    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:40:47.620142 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608310    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:40:47.620142 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608313    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:40:47.620142 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608315    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:40:47.620142 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608318    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:40:47.620142 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608321    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:40:47.620142 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608324    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:40:47.620142 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608326    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:40:47.620142 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608329    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:40:47.620142 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608331    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:40:47.620142 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608334    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:40:47.620589 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608336    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:40:47.620589 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608339    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:40:47.620589 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608342    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:40:47.620589 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608344    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:40:47.620589 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608347    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:40:47.620589 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608350    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:40:47.620589 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608352    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:40:47.620589 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608355    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:40:47.620589 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608357    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:40:47.620589 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608360    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:40:47.620589 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608363    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:40:47.620589 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608365    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:40:47.620589 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608370    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:40:47.620589 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608374    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:40:47.620589 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608376    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:40:47.620589 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608379    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:40:47.620589 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608381    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:40:47.620589 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608384    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:40:47.620589 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608387    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:40:47.620589 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608389    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:40:47.621086 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608392    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:40:47.621086 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608395    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:40:47.621086 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608398    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:40:47.621086 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608401    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:40:47.621086 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.608404    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:40:47.621086 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.615424    2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 00:40:47.621086 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.615443    2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 00:40:47.621086 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.615459    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 00:40:47.621086 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.615465    2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 00:40:47.621086 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615522    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:40:47.621086 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615528    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:40:47.621086 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615531    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:40:47.621086 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615534    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:40:47.621086 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615537    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:40:47.621086 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615539    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:40:47.621086 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615542    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:40:47.621086 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615545    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:40:47.621086 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615547    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:40:47.621086 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615550    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:40:47.621535 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615553    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:40:47.621535 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615556    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:40:47.621535 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615558    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:40:47.621535 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615561    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:40:47.621535 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615564    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:40:47.621535 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615567    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:40:47.621535 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615570    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:40:47.621535 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615573    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:40:47.621535 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615576    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:40:47.621535 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615578    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:40:47.621535 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615581    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:40:47.621535 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615584    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:40:47.621535 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615586    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:40:47.621535 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615589    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:40:47.621535 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615591    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:40:47.621535 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615594    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:40:47.621535 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615597    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:40:47.621535 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615600    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:40:47.621535 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615602    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:40:47.622015 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615606    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:40:47.622015 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615608    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:40:47.622015 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615611    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:40:47.622015 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615614    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:40:47.622015 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615616    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:40:47.622015 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615620    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:40:47.622015 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615622    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:40:47.622015 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615625    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:40:47.622015 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615628    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:40:47.622015 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615631    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:40:47.622015 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615633    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:40:47.622015 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615636    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:40:47.622015 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615639    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:40:47.622015 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615643    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:40:47.622015 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615646    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:40:47.622015 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615649    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:40:47.622015 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615652    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:40:47.622015 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615655    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:40:47.622015 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615658    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:40:47.622015 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615661    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:40:47.622494 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615664    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:40:47.622494 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615667    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:40:47.622494 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615671    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:40:47.622494 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615675    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:40:47.622494 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615678    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:40:47.622494 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615681    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:40:47.622494 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615684    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:40:47.622494 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615687    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:40:47.622494 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615689    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:40:47.622494 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615692    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:40:47.622494 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615695    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:40:47.622494 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615698    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:40:47.622494 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615701    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:40:47.622494 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615704    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:40:47.622494 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615707    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:40:47.622494 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615709    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:40:47.622494 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615712    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:40:47.622494 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615715    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:40:47.622494 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615718    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:40:47.622983 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615721    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:40:47.622983 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615723    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:40:47.622983 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615726    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:40:47.622983 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615728    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:40:47.622983 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615731    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:40:47.622983 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615734    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:40:47.622983 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615736    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:40:47.622983 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615739    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:40:47.622983 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615741    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:40:47.622983 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615744    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:40:47.622983 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615746    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:40:47.622983 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615749    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:40:47.622983 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615752    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:40:47.622983 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615754    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:40:47.622983 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615757    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:40:47.622983 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615760    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:40:47.622983 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615763    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:40:47.622983 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615765    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:40:47.622983 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615768    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:40:47.622983 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615770    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:40:47.623578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615773    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:40:47.623578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615775    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:40:47.623578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615778    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:40:47.623578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615780    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:40:47.623578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615783    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:40:47.623578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615786    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:40:47.623578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615788    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:40:47.623578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615791    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:40:47.623578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615794    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:40:47.623578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615797    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:40:47.623578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615799    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:40:47.623578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615802    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:40:47.623578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615805    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:40:47.623578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615808    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:40:47.623578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615811    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:40:47.623578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615813    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:40:47.623578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615816    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:40:47.623578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615818    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:40:47.623988 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.615844    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:40:47.623988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615992    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:40:47.623988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.615998    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:40:47.623988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616002    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:40:47.623988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616004    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:40:47.623988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616007    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:40:47.623988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616010    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:40:47.623988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616012    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:40:47.623988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616015    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:40:47.623988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616018    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:40:47.623988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616020    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:40:47.623988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616023    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:40:47.623988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616025    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:40:47.623988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616028    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:40:47.623988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616031    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:40:47.623988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616033    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:40:47.624386 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616036    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:40:47.624386 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616039    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:40:47.624386 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616041    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:40:47.624386 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616044    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:40:47.624386 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616047    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:40:47.624386 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616049    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:40:47.624386 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616052    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:40:47.624386 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616054    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:40:47.624386 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616057    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:40:47.624386 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616059    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:40:47.624386 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616062    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:40:47.624386 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616064    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:40:47.624386 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616067    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:40:47.624386 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616083    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:40:47.624386 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616086    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:40:47.624386 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616089    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:40:47.624386 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616091    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:40:47.624386 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616094    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:40:47.624386 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616096    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:40:47.624386 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616099    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:40:47.624862 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616102    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:40:47.624862 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616105    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:40:47.624862 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616108    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:40:47.624862 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616113    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:40:47.624862 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616117    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:40:47.624862 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616119    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:40:47.624862 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616123    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:40:47.624862 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616125    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:40:47.624862 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616128    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:40:47.624862 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616130    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:40:47.624862 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616133    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:40:47.624862 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616135    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:40:47.624862 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616138    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:40:47.624862 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616141    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:40:47.624862 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616144    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:40:47.624862 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616146    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:40:47.624862 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616149    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:40:47.624862 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616151    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:40:47.624862 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616154    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:40:47.625315 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616156    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:40:47.625315 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616159    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:40:47.625315 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616162    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:40:47.625315 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616164    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:40:47.625315 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616167    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:40:47.625315 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616169    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:40:47.625315 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616172    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:40:47.625315 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616175    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:40:47.625315 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616177    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:40:47.625315 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616180    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:40:47.625315 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616183    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:40:47.625315 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616188    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:40:47.625315 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616190    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:40:47.625315 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616193    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:40:47.625315 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616195    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:40:47.625315 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616198    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:40:47.625315 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616200    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:40:47.625315 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616204    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:40:47.625315 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616206    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:40:47.625763 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616209    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:40:47.625763 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616212    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:40:47.625763 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616214    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:40:47.625763 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616217    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:40:47.625763 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616219    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:40:47.625763 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616222    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:40:47.625763 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616225    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:40:47.625763 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616228    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:40:47.625763 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616231    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:40:47.625763 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616233    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:40:47.625763 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616236    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:40:47.625763 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616238    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:40:47.625763 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616241    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:40:47.625763 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616244    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:40:47.625763 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616247    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:40:47.625763 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616249    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:40:47.625763 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616252    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:40:47.625763 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616254    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:40:47.625763 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616257    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:40:47.625763 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616260    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:40:47.626239 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616262    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:40:47.626239 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616265    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:40:47.626239 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616267    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:40:47.626239 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616270    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:40:47.626239 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616275    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:40:47.626239 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616279    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:40:47.626239 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616282    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:40:47.626239 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616284    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:40:47.626239 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616287    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:40:47.626239 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616289    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:40:47.626239 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616292    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:40:47.626239 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616295    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:40:47.626239 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616298    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:40:47.626239 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.616322    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:40:47.626239 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616355    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:40:47.626578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616360    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:40:47.626578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616363    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:40:47.626578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616366    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:40:47.626578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616368    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:40:47.626578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616371    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:40:47.626578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616374    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:40:47.626578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616376    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:40:47.626578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616379    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:40:47.626578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616381    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:40:47.626578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616384    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:40:47.626578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616388    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:40:47.626578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616392    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:40:47.626578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616396    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:40:47.626578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616399    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:40:47.626578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616402    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:40:47.626578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616405    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:40:47.626578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616407    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:40:47.626578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616410    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:40:47.626578 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616413    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:40:47.627046 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616416    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:40:47.627046 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616418    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:40:47.627046 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616422    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:40:47.627046 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616425    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:40:47.627046 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616427    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:40:47.627046 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616430    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:40:47.627046 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616432    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:40:47.627046 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616435    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:40:47.627046 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616437    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:40:47.627046 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616440    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:40:47.627046 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616443    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:40:47.627046 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616446    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:40:47.627046 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616448    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:40:47.627046 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616453    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:40:47.627046 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616455    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:40:47.627046 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616458    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:40:47.627046 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616461    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:40:47.627046 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616463    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:40:47.627046 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616466    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:40:47.627046 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616468    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:40:47.627525 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616471    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:40:47.627525 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616474    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:40:47.627525 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616476    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:40:47.627525 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616479    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:40:47.627525 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616481    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:40:47.627525 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616484    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:40:47.627525 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616486    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:40:47.627525 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616489    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:40:47.627525 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616491    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:40:47.627525 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616494    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:40:47.627525 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616497    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:40:47.627525 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616500    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:40:47.627525 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616502    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:40:47.627525 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616505    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:40:47.627525 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616509    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:40:47.627525 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616512    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:40:47.627525 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616514    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:40:47.627525 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616518    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:40:47.627525 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616522    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:40:47.627525 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616525    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:40:47.627988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616527    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:40:47.627988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616531    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:40:47.627988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616534    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:40:47.627988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616537    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:40:47.627988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616540    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:40:47.627988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616546    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:40:47.627988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616549    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:40:47.627988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616552    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:40:47.627988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616555    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:40:47.627988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616558    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:40:47.627988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616560    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:40:47.627988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616563    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:40:47.627988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616566    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:40:47.627988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616568    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:40:47.627988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616571    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:40:47.627988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616573    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:40:47.627988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616576    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:40:47.627988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616578    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:40:47.627988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616581    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:40:47.627988 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616584    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:40:47.628469 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616587    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:40:47.628469 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616589    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:40:47.628469 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616592    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:40:47.628469 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616594    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:40:47.628469 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616597    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:40:47.628469 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616600    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:40:47.628469 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616603    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:40:47.628469 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616606    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:40:47.628469 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616609    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:40:47.628469 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616613    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:40:47.628469 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616616    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:40:47.628469 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616618    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:40:47.628469 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616621    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:40:47.628469 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616624    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:40:47.628469 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616626    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:40:47.628469 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616629    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:40:47.628469 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616632    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:40:47.628469 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616636    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:40:47.628469 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616639    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:40:47.628985 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616641    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:40:47.628985 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616644    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:40:47.628985 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616646    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:40:47.628985 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616649    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:40:47.628985 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616652    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:40:47.628985 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616654    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:40:47.628985 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:47.616657    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:40:47.628985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.616737    2582 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 00:40:47.628985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.619557    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 00:40:47.628985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.620906    2582 server.go:1014] "Starting client certificate rotation"
Oct 30 00:40:47.628985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.621010    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 00:40:47.628985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.621045    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 00:40:47.628985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.623336    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:40:47.628985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.624757    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:40:47.634615 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.634599    2582 log.go:25] "Validated CRI v1 runtime API"
Oct 30 00:40:47.637787 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.637773    2582 log.go:25] "Validated CRI v1 image API"
Oct 30 00:40:47.638566 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.638540    2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 00:40:47.640877 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.640858    2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ef999e16-7a0f-4eab-9b2c-4e9ab340269a:/dev/nvme0n1p3 f6480457-aeeb-4a23-830e-3c364a92c5fe:/dev/nvme0n1p4]
Oct 30 00:40:47.640936 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.640876    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 00:40:47.647237 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.647106    2582 manager.go:217] Machine: {Timestamp:2026-10-30 00:40:47.645700814 +0000 UTC m=+0.337961085 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099947 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec200a57591a5ecf36a35b4f290abc7e SystemUUID:ec200a57-591a-5ecf-36a3-5b4f290abc7e BootID:0744abf2-5a97-4d46-b8a1-995f2b472c34 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d8:17:b3:6b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d8:17:b3:6b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:fa:01:74:2b:3c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 00:40:47.647237 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.647225    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 00:40:47.647408 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.647314    2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 00:40:47.650321 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.650298    2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 00:40:47.650491 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.650321    2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-210.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 00:40:47.650536 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.650494    2582 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 00:40:47.650536 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.650503    2582 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 00:40:47.650536 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.650534    2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:40:47.650613 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.650553    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:40:47.651397 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.651387    2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 00:40:47.655005 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.654993    2582 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 00:40:47.655138 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.655129    2582 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 00:40:47.655870 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.655862    2582 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 00:40:47.655906 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.655874    2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 00:40:47.655906 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.655884    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 00:40:47.655906 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.655895    2582 kubelet.go:405] "Adding apiserver pod source"
Oct 30 00:40:47.655906 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.655904    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 00:40:47.656265 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.656254    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:40:47.658670 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.658653    2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 00:40:47.663508 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.663495    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:40:47.664224 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.664212    2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 00:40:47.664279 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.664243    2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 00:40:47.664427 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.664417    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 00:40:47.664456 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.664433    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 00:40:47.664456 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.664440    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 00:40:47.664456 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.664446    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 00:40:47.664456 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.664451    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 00:40:47.664555 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.664458    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 00:40:47.665058 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.665050    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 00:40:47.665098 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.665059    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 00:40:47.665098 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.665067    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 00:40:47.665098 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.665085    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 00:40:47.665098 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.665094    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 00:40:47.665299 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.665102    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 00:40:47.665563 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.665553    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 00:40:47.665595 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.665567    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 00:40:47.670644 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.670631    2582 server.go:1263] "Started kubelet"
Oct 30 00:40:47.670795 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.670740    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 00:40:47.670847 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.670820    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 00:40:47.671115 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.671093    2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 00:40:47.671505 ip-10-0-134-210 systemd[1]: Started Kubernetes Kubelet.
Oct 30 00:40:47.671974 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.671956    2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 00:40:47.673484 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.673468    2582 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 00:40:47.676451 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:47.676428    2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 00:40:47.677502 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.677481    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 00:40:47.678067 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.678049    2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 00:40:47.678213 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:47.678176    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-210.ec2.internal\" not found"
Oct 30 00:40:47.678882 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.678858    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 00:40:47.678882 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.678859    2582 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 00:40:47.678882 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.678884    2582 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 00:40:47.679266 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.679252    2582 factory.go:55] Registering systemd factory
Oct 30 00:40:47.679323 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.679291    2582 factory.go:223] Registration of the systemd container factory successfully
Oct 30 00:40:47.679425 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.679413    2582 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 00:40:47.679425 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.679424    2582 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 00:40:47.679527 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.679514    2582 factory.go:153] Registering CRI-O factory
Oct 30 00:40:47.679559 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.679532    2582 factory.go:223] Registration of the crio container factory successfully
Oct 30 00:40:47.679599 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.679589    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 00:40:47.679632 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.679612    2582 factory.go:103] Registering Raw factory
Oct 30 00:40:47.679632 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.679627    2582 manager.go:1199] Started watching for new ooms in manager
Oct 30 00:40:47.680533 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.680520    2582 manager.go:319] Starting recovery of all containers
Oct 30 00:40:47.691309 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.691286    2582 manager.go:324] Recovery completed
Oct 30 00:40:47.694157 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.694118    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 00:40:47.697290 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.697240    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:40:47.699322 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.699300    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:40:47.699412 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.699338    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:40:47.699412 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.699352    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:40:47.699947 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.699931    2582 cpu_manager.go:226] "Starting" policy="none"
Oct 30 00:40:47.699947 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.699943    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:40:47.700114 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.699945    2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 00:40:47.700114 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.699987    2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 00:40:47.700651 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.700635    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:40:47.700705 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.700666    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:40:47.700705 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.700681    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:40:47.702236 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.702221    2582 policy_none.go:50] "Start"
Oct 30 00:40:47.702317 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.702240    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 00:40:47.702317 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.702253    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 00:40:47.704348 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.704334    2582 policy_none.go:44] "Start"
Oct 30 00:40:47.708221 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:47.708190    2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-210.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 00:40:47.708341 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.708205    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-210.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 00:40:47.709252 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:47.708196    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-210.ec2.internal.18e327cf3db55703  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-210.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-210.ec2.internal,},FirstTimestamp:2026-10-30 00:40:47.670597379 +0000 UTC m=+0.362857651,LastTimestamp:2026-10-30 00:40:47.670597379 +0000 UTC m=+0.362857651,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-210.ec2.internal,}"
Oct 30 00:40:47.721158 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:47.721137    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-210.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 00:40:47.721677 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:47.721645    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 00:40:47.721677 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:47.721654    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 00:40:47.721826 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.721716    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 00:40:47.743760 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.743730    2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 00:40:47.746942 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.746929    2582 manager.go:341] "Starting Device Plugin manager"
Oct 30 00:40:47.747034 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:47.746955    2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 00:40:47.747034 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.746964    2582 server.go:87] "Starting device plugin registration server"
Oct 30 00:40:47.747247 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.747234    2582 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 00:40:47.747294 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.747248    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 00:40:47.747348 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.747338    2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 00:40:47.747416 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.747404    2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 00:40:47.747416 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.747416    2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 00:40:47.747896 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:47.747867    2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 00:40:47.747964 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:47.747919    2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-210.ec2.internal\" not found"
Oct 30 00:40:47.748356 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.748341    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4l2ll"
Oct 30 00:40:47.758811 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.758795    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4l2ll"
Oct 30 00:40:47.760884 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:47.760817    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-210.ec2.internal.18e327cf3f6b9f10  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-210.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-210.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-210.ec2.internal,},FirstTimestamp:2026-10-30 00:40:47.699320592 +0000 UTC m=+0.391580871,LastTimestamp:2026-10-30 00:40:47.699320592 +0000 UTC m=+0.391580871,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-210.ec2.internal,}"
Oct 30 00:40:47.835629 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.834281    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 00:40:47.835629 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.834307    2582 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 00:40:47.835629 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.834328    2582 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 00:40:47.835629 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:47.834364    2582 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 00:40:47.847508 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.847489    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:40:47.848305 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.848259    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:40:47.848305 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.848291    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:40:47.848443 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.848306    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:40:47.848443 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.848337    2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-210.ec2.internal"
Oct 30 00:40:47.850531 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.850507    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:40:47.868622 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.868472    2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-210.ec2.internal"
Oct 30 00:40:47.868713 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:47.868628    2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-210.ec2.internal\": node \"ip-10-0-134-210.ec2.internal\" not found"
Oct 30 00:40:47.909565 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:47.909540    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-210.ec2.internal\" not found"
Oct 30 00:40:47.934493 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.934463    2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-210.ec2.internal"]
Oct 30 00:40:47.934574 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.934526    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:40:47.935427 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.935413    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:40:47.935492 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.935441    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:40:47.935492 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.935450    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:40:47.937586 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.937575    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:40:47.937761 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.937748    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal"
Oct 30 00:40:47.937799 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.937794    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:40:47.938282 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.938264    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:40:47.938282 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.938275    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:40:47.938411 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.938293    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:40:47.938411 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.938299    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:40:47.938411 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.938307    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:40:47.938411 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.938314    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:40:47.940930 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.940908    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-210.ec2.internal"
Oct 30 00:40:47.941034 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.940945    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:40:47.941611 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.941591    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:40:47.941700 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.941623    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:40:47.941700 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.941637    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:40:47.979670 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.979654    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e4c9778b041df41465cbcba4cba59e17-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal\" (UID: \"e4c9778b041df41465cbcba4cba59e17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal"
Oct 30 00:40:47.979756 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.979677    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4c9778b041df41465cbcba4cba59e17-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal\" (UID: \"e4c9778b041df41465cbcba4cba59e17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal"
Oct 30 00:40:47.979756 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:47.979696    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17edaf9736d7ee1e0409a6d573ccb434-config\") pod \"kube-apiserver-proxy-ip-10-0-134-210.ec2.internal\" (UID: \"17edaf9736d7ee1e0409a6d573ccb434\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-210.ec2.internal"
Oct 30 00:40:47.980794 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:47.980775    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-210.ec2.internal\" not found" node="ip-10-0-134-210.ec2.internal"
Oct 30 00:40:47.983194 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:47.983176    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-210.ec2.internal\" not found" node="ip-10-0-134-210.ec2.internal"
Oct 30 00:40:48.009668 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:48.009645    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-210.ec2.internal\" not found"
Oct 30 00:40:48.080742 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:48.080718    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e4c9778b041df41465cbcba4cba59e17-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal\" (UID: \"e4c9778b041df41465cbcba4cba59e17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal"
Oct 30 00:40:48.080854 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:48.080744    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4c9778b041df41465cbcba4cba59e17-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal\" (UID: \"e4c9778b041df41465cbcba4cba59e17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal"
Oct 30 00:40:48.080854 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:48.080761    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17edaf9736d7ee1e0409a6d573ccb434-config\") pod \"kube-apiserver-proxy-ip-10-0-134-210.ec2.internal\" (UID: \"17edaf9736d7ee1e0409a6d573ccb434\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-210.ec2.internal"
Oct 30 00:40:48.080854 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:48.080811    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17edaf9736d7ee1e0409a6d573ccb434-config\") pod \"kube-apiserver-proxy-ip-10-0-134-210.ec2.internal\" (UID: \"17edaf9736d7ee1e0409a6d573ccb434\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-210.ec2.internal"
Oct 30 00:40:48.080854 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:48.080830    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4c9778b041df41465cbcba4cba59e17-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal\" (UID: \"e4c9778b041df41465cbcba4cba59e17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal"
Oct 30 00:40:48.080993 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:48.080832    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e4c9778b041df41465cbcba4cba59e17-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal\" (UID: \"e4c9778b041df41465cbcba4cba59e17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal"
Oct 30 00:40:48.110150 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:48.110052    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-210.ec2.internal\" not found"
Oct 30 00:40:48.210376 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:48.210342    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-210.ec2.internal\" not found"
Oct 30 00:40:48.284273 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:48.284246    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal"
Oct 30 00:40:48.286742 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:48.286724    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-210.ec2.internal"
Oct 30 00:40:48.310920 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:48.310889    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-210.ec2.internal\" not found"
Oct 30 00:40:48.411404 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:48.411315    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-210.ec2.internal\" not found"
Oct 30 00:40:48.511737 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:48.511701    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-210.ec2.internal\" not found"
Oct 30 00:40:48.612605 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:48.612574    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-210.ec2.internal\" not found"
Oct 30 00:40:48.620878 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:48.620843    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 00:40:48.621107 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:48.621067    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 00:40:48.713008 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:48.712980    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-210.ec2.internal\" not found"
Oct 30 00:40:48.716783 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:48.716764    2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:40:48.764086 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:48.764014    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:35:48 +0000 UTC" deadline="2028-06-14 06:50:52.111234308 +0000 UTC"
Oct 30 00:40:48.764086 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:48.764060    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14238h10m3.347179502s"
Oct 30 00:40:48.779252 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:48.779222    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal"
Oct 30 00:40:48.808306 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:48.808278    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:40:48.810339 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:48.810315    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-210.ec2.internal"
Oct 30 00:40:48.834608 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:48.834584    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:40:48.861017 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:48.860979    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17edaf9736d7ee1e0409a6d573ccb434.slice/crio-e00c480be6b0794693cf776efc17b9266d72737419c9ab17343e8f19abe31439 WatchSource:0}: Error finding container e00c480be6b0794693cf776efc17b9266d72737419c9ab17343e8f19abe31439: Status 404 returned error can't find the container with id e00c480be6b0794693cf776efc17b9266d72737419c9ab17343e8f19abe31439
Oct 30 00:40:48.861940 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:48.861916    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c9778b041df41465cbcba4cba59e17.slice/crio-6ff453ef83e8b4374463dbceb3eda86c51c7b10b2b23e81a7aabffa9367d65c2 WatchSource:0}: Error finding container 6ff453ef83e8b4374463dbceb3eda86c51c7b10b2b23e81a7aabffa9367d65c2: Status 404 returned error can't find the container with id 6ff453ef83e8b4374463dbceb3eda86c51c7b10b2b23e81a7aabffa9367d65c2
Oct 30 00:40:48.866654 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:48.866639    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:40:49.097349 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.097269    2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:40:49.182240 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.182204    2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:40:49.657910 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.657881    2582 apiserver.go:52] "Watching apiserver"
Oct 30 00:40:49.678991 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.678964    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 00:40:49.708046 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.708018    2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 00:40:49.708237 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.708190    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:40:49.708427 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.708406    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-tz5vn","openshift-network-diagnostics/network-check-target-jh728","openshift-network-operator/iptables-alerter-ljlrc","kube-system/kube-apiserver-proxy-ip-10-0-134-210.ec2.internal","openshift-multus/network-metrics-daemon-b4686","kube-system/global-pull-secret-syncer-wk65n","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn","openshift-dns/node-resolver-49rdp","openshift-image-registry/node-ca-qdjmq","openshift-cluster-node-tuning-operator/tuned-q4bch","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal","openshift-multus/multus-qp98l","openshift-ovn-kubernetes/ovnkube-node-7855k","kube-system/konnectivity-agent-77dcb"]
Oct 30 00:40:49.711082 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.711037    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.713648 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.713612    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.716480 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.716460    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ljlrc"
Oct 30 00:40:49.718722 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.718700    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-77dcb"
Oct 30 00:40:49.721407 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.721378    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:40:49.721520 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:49.721460    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b4686" podUID="0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493"
Oct 30 00:40:49.728340 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.728312    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:40:49.728457 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:49.728397    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wk65n" podUID="bbef968a-7aa4-4e23-9278-3706e60d8c0b"
Oct 30 00:40:49.736800 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.736775    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qdjmq"
Oct 30 00:40:49.738348 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.738237    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:40:49.738348 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.738174    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 00:40:49.739942 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.739919    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:40:49.743978 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.743348    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:40:49.743978 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:49.743427    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh728" podUID="204212cc-1c4c-4bfb-8b41-03a98042166c"
Oct 30 00:40:49.745495 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.745474    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:40:49.748282 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.748258    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-49rdp"
Oct 30 00:40:49.750225 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.750207    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-92qkq\""
Oct 30 00:40:49.752913 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.752893    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.772311 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.772286    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 00:40:49.772416 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.772360    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-flvdk"
Oct 30 00:40:49.780673 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.780652    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 00:40:49.789561 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.789539    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5fade25-18f0-4e3f-978a-5bb39a356ab0-system-cni-dir\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:40:49.789684 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.789568    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-multus-socket-dir-parent\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.789684 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.789586    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-var-lib-cni-multus\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.789684 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.789604    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-modprobe-d\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.789684 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.789629    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-lib-modules\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.789893 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.789719    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f5fade25-18f0-4e3f-978a-5bb39a356ab0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:40:49.789893 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.789763    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f5fade25-18f0-4e3f-978a-5bb39a356ab0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:40:49.789893 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.789797    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-run-systemd\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.789893 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.789829    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j89wv\" (UniqueName: \"kubernetes.io/projected/380df728-1bf4-4258-ba04-c31e1d28a3aa-kube-api-access-j89wv\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.790131 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.789909    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwnlf\" (UniqueName: \"kubernetes.io/projected/4770276e-e9dc-4f6c-94f6-d2dfe2f6f115-kube-api-access-lwnlf\") pod \"node-resolver-49rdp\" (UID: \"4770276e-e9dc-4f6c-94f6-d2dfe2f6f115\") " pod="openshift-dns/node-resolver-49rdp"
Oct 30 00:40:49.790131 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.789957    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-log-socket\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.790131 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.789986    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.790131 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790030    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/380df728-1bf4-4258-ba04-c31e1d28a3aa-env-overrides\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.790131 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790060    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-etc-openvswitch\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.790131 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790122    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/380df728-1bf4-4258-ba04-c31e1d28a3aa-ovnkube-script-lib\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.790443 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790158    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbef968a-7aa4-4e23-9278-3706e60d8c0b-kubelet-config\") pod \"global-pull-secret-syncer-wk65n\" (UID: \"bbef968a-7aa4-4e23-9278-3706e60d8c0b\") " pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:40:49.790443 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790219    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-socket-dir\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:40:49.790443 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790255    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-systemd\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.790443 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790282    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-slash\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.790443 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790323    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-run-ovn\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.790443 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790365    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk7mm\" (UniqueName: \"kubernetes.io/projected/f5fade25-18f0-4e3f-978a-5bb39a356ab0-kube-api-access-tk7mm\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:40:49.790443 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790400    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-hostroot\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.790443 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790424    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnf9q\" (UniqueName: \"kubernetes.io/projected/0eac4baa-d0d9-43c0-8f81-3733a026e091-kube-api-access-nnf9q\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.790854 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790471    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-systemd-units\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.790854 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790505    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-node-log\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.790854 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790546    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:40:49.790854 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790591    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5fade25-18f0-4e3f-978a-5bb39a356ab0-cni-binary-copy\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:40:49.790854 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790627    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kksvb\" (UniqueName: \"kubernetes.io/projected/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-kube-api-access-kksvb\") pod \"network-metrics-daemon-b4686\" (UID: \"0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493\") " pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:40:49.790854 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790653    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0eac4baa-d0d9-43c0-8f81-3733a026e091-cni-binary-copy\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.790854 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790703    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-sysctl-d\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.790854 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790739    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-run-netns\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.790854 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790765    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nkxz\" (UniqueName: \"kubernetes.io/projected/19a807d5-c007-47b4-953c-28de606ababc-kube-api-access-8nkxz\") pod \"node-ca-qdjmq\" (UID: \"19a807d5-c007-47b4-953c-28de606ababc\") " pod="openshift-image-registry/node-ca-qdjmq"
Oct 30 00:40:49.790854 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790810    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4770276e-e9dc-4f6c-94f6-d2dfe2f6f115-hosts-file\") pod \"node-resolver-49rdp\" (UID: \"4770276e-e9dc-4f6c-94f6-d2dfe2f6f115\") " pod="openshift-dns/node-resolver-49rdp"
Oct 30 00:40:49.790854 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790835    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ac86565-2bfb-45e4-840d-c0f3a29cdfa5-agent-certs\") pod \"konnectivity-agent-77dcb\" (UID: \"2ac86565-2bfb-45e4-840d-c0f3a29cdfa5\") " pod="kube-system/konnectivity-agent-77dcb"
Oct 30 00:40:49.791441 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790860    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ac86565-2bfb-45e4-840d-c0f3a29cdfa5-konnectivity-ca\") pod \"konnectivity-agent-77dcb\" (UID: \"2ac86565-2bfb-45e4-840d-c0f3a29cdfa5\") " pod="kube-system/konnectivity-agent-77dcb"
Oct 30 00:40:49.791441 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790888    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae0f3040-049e-4f4e-a6e5-206b31aeea55-tmp\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.791441 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790924    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-cni-netd\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.791441 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.790947    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/19a807d5-c007-47b4-953c-28de606ababc-serviceca\") pod \"node-ca-qdjmq\" (UID: \"19a807d5-c007-47b4-953c-28de606ababc\") " pod="openshift-image-registry/node-ca-qdjmq"
Oct 30 00:40:49.791441 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791010    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-run-k8s-cni-cncf-io\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.791441 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791059    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq6ls\" (UniqueName: \"kubernetes.io/projected/c75bc9c9-74be-412b-a5ac-794f971bea31-kube-api-access-jq6ls\") pod \"iptables-alerter-ljlrc\" (UID: \"c75bc9c9-74be-412b-a5ac-794f971bea31\") " pod="openshift-network-operator/iptables-alerter-ljlrc"
Oct 30 00:40:49.791441 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791087    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-registration-dir\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:40:49.791441 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791113    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpn92\" (UniqueName: \"kubernetes.io/projected/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-kube-api-access-bpn92\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:40:49.791441 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791161    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/380df728-1bf4-4258-ba04-c31e1d28a3aa-ovn-node-metrics-cert\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.791441 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791191    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbef968a-7aa4-4e23-9278-3706e60d8c0b-dbus\") pod \"global-pull-secret-syncer-wk65n\" (UID: \"bbef968a-7aa4-4e23-9278-3706e60d8c0b\") " pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:40:49.791441 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791226    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-multus-cni-dir\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.791441 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791250    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-host\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.791441 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791279    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpqz6\" (UniqueName: \"kubernetes.io/projected/ae0f3040-049e-4f4e-a6e5-206b31aeea55-kube-api-access-jpqz6\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.791441 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791305    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0eac4baa-d0d9-43c0-8f81-3733a026e091-multus-daemon-config\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.791441 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791343    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-var-lib-kubelet\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.791441 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791433    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-cni-bin\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.792282 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791473    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-etc-selinux\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:40:49.792282 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791502    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5fade25-18f0-4e3f-978a-5bb39a356ab0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:40:49.792282 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791531    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs\") pod \"network-metrics-daemon-b4686\" (UID: \"0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493\") " pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:40:49.792282 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791558    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-run-netns\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.792282 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791586    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-etc-kubernetes\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.792282 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791619    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c75bc9c9-74be-412b-a5ac-794f971bea31-host-slash\") pod \"iptables-alerter-ljlrc\" (UID: \"c75bc9c9-74be-412b-a5ac-794f971bea31\") " pod="openshift-network-operator/iptables-alerter-ljlrc"
Oct 30 00:40:49.792282 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791659    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-multus-conf-dir\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.792282 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791683    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-kubelet\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.792282 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791709    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-var-lib-openvswitch\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.792282 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791726    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-run-ovn-kubernetes\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.792282 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791792    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbef968a-7aa4-4e23-9278-3706e60d8c0b-original-pull-secret\") pod \"global-pull-secret-syncer-wk65n\" (UID: \"bbef968a-7aa4-4e23-9278-3706e60d8c0b\") " pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:40:49.792282 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791822    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-sys-fs\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:40:49.792282 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791842    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-run-openvswitch\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.792282 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791868    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/380df728-1bf4-4258-ba04-c31e1d28a3aa-ovnkube-config\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.792282 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791890    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19a807d5-c007-47b4-953c-28de606ababc-host\") pod \"node-ca-qdjmq\" (UID: \"19a807d5-c007-47b4-953c-28de606ababc\") " pod="openshift-image-registry/node-ca-qdjmq"
Oct 30 00:40:49.792282 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791916    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-device-dir\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:40:49.793101 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791940    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-run\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.793101 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.791970    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-var-lib-cni-bin\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.793101 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.792000    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-run-multus-certs\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.793101 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.792027    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-sysconfig\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.793101 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.792054    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-kubernetes\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.793101 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.792143    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-tuned\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.793101 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.792191    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5fade25-18f0-4e3f-978a-5bb39a356ab0-cnibin\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:40:49.793101 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.792209    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c75bc9c9-74be-412b-a5ac-794f971bea31-iptables-alerter-script\") pod \"iptables-alerter-ljlrc\" (UID: \"c75bc9c9-74be-412b-a5ac-794f971bea31\") " pod="openshift-network-operator/iptables-alerter-ljlrc"
Oct 30 00:40:49.793101 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.792229    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-var-lib-kubelet\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.793101 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.792251    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-sysctl-conf\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.793101 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.792272    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-sys\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.793101 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.792297    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-system-cni-dir\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.793101 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.792322    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-cnibin\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.793101 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.792348    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vvj2\" (UniqueName: \"kubernetes.io/projected/204212cc-1c4c-4bfb-8b41-03a98042166c-kube-api-access-7vvj2\") pod \"network-check-target-jh728\" (UID: \"204212cc-1c4c-4bfb-8b41-03a98042166c\") " pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:40:49.793101 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.792374    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4770276e-e9dc-4f6c-94f6-d2dfe2f6f115-tmp-dir\") pod \"node-resolver-49rdp\" (UID: \"4770276e-e9dc-4f6c-94f6-d2dfe2f6f115\") " pod="openshift-dns/node-resolver-49rdp"
Oct 30 00:40:49.838933 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.838902    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 00:40:49.839107 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.838950    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 00:40:49.839107 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.839067    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 00:40:49.839255 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.839113    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 00:40:49.839316 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.839304    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 00:40:49.839433 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.839417    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 00:40:49.839496 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.839444    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 00:40:49.839496 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.839458    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 00:40:49.839496 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.839479    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:40:49.839496 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.839459    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-210.ec2.internal" event={"ID":"17edaf9736d7ee1e0409a6d573ccb434","Type":"ContainerStarted","Data":"e00c480be6b0794693cf776efc17b9266d72737419c9ab17343e8f19abe31439"}
Oct 30 00:40:49.839692 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.839524    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 00:40:49.839692 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.839639    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 00:40:49.839692 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.839668    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 00:40:49.839692 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.839680    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 00:40:49.839692 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.839687    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 00:40:49.839903 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.839683    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 00:40:49.839903 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.839748    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 00:40:49.839903 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.839765    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 00:40:49.839903 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.839780    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 00:40:49.839903 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.839799    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 00:40:49.839903 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.839826    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 00:40:49.840182 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.840066    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 00:40:49.840182 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.840134    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 00:40:49.841203 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.841171    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal" event={"ID":"e4c9778b041df41465cbcba4cba59e17","Type":"ContainerStarted","Data":"6ff453ef83e8b4374463dbceb3eda86c51c7b10b2b23e81a7aabffa9367d65c2"}
Oct 30 00:40:49.854947 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.854910    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 00:40:49.855110 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.854964    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5b9kk\""
Oct 30 00:40:49.855110 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.855008    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z5p7p\""
Oct 30 00:40:49.855110 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.855019    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jgsn4\""
Oct 30 00:40:49.855110 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.855065    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7sn5q\""
Oct 30 00:40:49.855398 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.855386    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cg6zp\""
Oct 30 00:40:49.855458 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.855444    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-szv55\""
Oct 30 00:40:49.855513 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.855453    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7htjs\""
Oct 30 00:40:49.855563 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.855525    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 00:40:49.855612 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.855576    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f5gxs\""
Oct 30 00:40:49.855612 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.855576    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 00:40:49.893512 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.893465    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpn92\" (UniqueName: \"kubernetes.io/projected/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-kube-api-access-bpn92\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:40:49.893691 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.893519    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/380df728-1bf4-4258-ba04-c31e1d28a3aa-ovn-node-metrics-cert\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.893691 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.893557    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbef968a-7aa4-4e23-9278-3706e60d8c0b-dbus\") pod \"global-pull-secret-syncer-wk65n\" (UID: \"bbef968a-7aa4-4e23-9278-3706e60d8c0b\") " pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:40:49.893691 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.893582    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-multus-cni-dir\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.893691 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.893607    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-host\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.893691 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.893647    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpqz6\" (UniqueName: \"kubernetes.io/projected/ae0f3040-049e-4f4e-a6e5-206b31aeea55-kube-api-access-jpqz6\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.893691 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.893674    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0eac4baa-d0d9-43c0-8f81-3733a026e091-multus-daemon-config\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.894003 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.893698    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-var-lib-kubelet\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.894003 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.893722    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-cni-bin\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.894003 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.893747    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-etc-selinux\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:40:49.894003 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.893770    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5fade25-18f0-4e3f-978a-5bb39a356ab0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:40:49.894003 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.893793    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs\") pod \"network-metrics-daemon-b4686\" (UID: \"0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493\") " pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:40:49.894003 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.893816    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-run-netns\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.894003 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.893851    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-etc-kubernetes\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.894003 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.893883    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c75bc9c9-74be-412b-a5ac-794f971bea31-host-slash\") pod \"iptables-alerter-ljlrc\" (UID: \"c75bc9c9-74be-412b-a5ac-794f971bea31\") " pod="openshift-network-operator/iptables-alerter-ljlrc"
Oct 30 00:40:49.894003 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.893893    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-cni-bin\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.894003 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.893913    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-multus-conf-dir\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.894003 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.893893    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbef968a-7aa4-4e23-9278-3706e60d8c0b-dbus\") pod \"global-pull-secret-syncer-wk65n\" (UID: \"bbef968a-7aa4-4e23-9278-3706e60d8c0b\") " pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:40:49.894003 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.893941    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-kubelet\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.894003 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.893967    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-var-lib-openvswitch\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.894003 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:49.893988    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:40:49.894679 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.893990    2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 00:40:49.894679 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894039    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-run-ovn-kubernetes\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.894679 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:49.894072    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs podName:0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493 nodeName:}" failed. No retries permitted until 2026-10-30 00:40:50.39404573 +0000 UTC m=+3.086306011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs") pod "network-metrics-daemon-b4686" (UID: "0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:40:49.894679 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894086    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-run-netns\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.894679 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894128    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-etc-kubernetes\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.894679 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894195    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c75bc9c9-74be-412b-a5ac-794f971bea31-host-slash\") pod \"iptables-alerter-ljlrc\" (UID: \"c75bc9c9-74be-412b-a5ac-794f971bea31\") " pod="openshift-network-operator/iptables-alerter-ljlrc"
Oct 30 00:40:49.894679 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894249    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-multus-conf-dir\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.894679 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894256    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-var-lib-kubelet\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.894679 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894299    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-kubelet\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.894679 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894187    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-etc-selinux\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:40:49.894679 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.893994    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-run-ovn-kubernetes\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.894679 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894341    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5fade25-18f0-4e3f-978a-5bb39a356ab0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:40:49.894679 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894348    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-var-lib-openvswitch\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.894679 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894364    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbef968a-7aa4-4e23-9278-3706e60d8c0b-original-pull-secret\") pod \"global-pull-secret-syncer-wk65n\" (UID: \"bbef968a-7aa4-4e23-9278-3706e60d8c0b\") " pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:40:49.894679 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894400    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-sys-fs\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:40:49.894679 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894428    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-run-openvswitch\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.894679 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894455    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/380df728-1bf4-4258-ba04-c31e1d28a3aa-ovnkube-config\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.895496 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:49.894480    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:40:49.895496 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894403    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-host\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.895496 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894525    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19a807d5-c007-47b4-953c-28de606ababc-host\") pod \"node-ca-qdjmq\" (UID: \"19a807d5-c007-47b4-953c-28de606ababc\") " pod="openshift-image-registry/node-ca-qdjmq"
Oct 30 00:40:49.895496 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:49.894537    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbef968a-7aa4-4e23-9278-3706e60d8c0b-original-pull-secret podName:bbef968a-7aa4-4e23-9278-3706e60d8c0b nodeName:}" failed. No retries permitted until 2026-10-30 00:40:50.39452062 +0000 UTC m=+3.086780884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbef968a-7aa4-4e23-9278-3706e60d8c0b-original-pull-secret") pod "global-pull-secret-syncer-wk65n" (UID: "bbef968a-7aa4-4e23-9278-3706e60d8c0b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:40:49.895496 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894481    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19a807d5-c007-47b4-953c-28de606ababc-host\") pod \"node-ca-qdjmq\" (UID: \"19a807d5-c007-47b4-953c-28de606ababc\") " pod="openshift-image-registry/node-ca-qdjmq"
Oct 30 00:40:49.895496 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894559    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-sys-fs\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:40:49.895496 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894574    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-run-openvswitch\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.895496 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894582    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-device-dir\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:40:49.895496 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894607    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-run\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.895496 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894642    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-var-lib-cni-bin\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.895496 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894666    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-run-multus-certs\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.895496 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894690    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-sysconfig\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.895496 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894700    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-run\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.895496 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894715    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-kubernetes\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.895496 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894755    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-tuned\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.895496 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894768    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-device-dir\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:40:49.895496 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894782    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5fade25-18f0-4e3f-978a-5bb39a356ab0-cnibin\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:40:49.895985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894808    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c75bc9c9-74be-412b-a5ac-794f971bea31-iptables-alerter-script\") pod \"iptables-alerter-ljlrc\" (UID: \"c75bc9c9-74be-412b-a5ac-794f971bea31\") " pod="openshift-network-operator/iptables-alerter-ljlrc"
Oct 30 00:40:49.895985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894821    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-sysconfig\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.895985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894834    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-var-lib-kubelet\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.895985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894859    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-sysctl-conf\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.895985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894862    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-var-lib-cni-bin\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.895985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894884    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-sys\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.895985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894890    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0eac4baa-d0d9-43c0-8f81-3733a026e091-multus-daemon-config\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.895985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894903    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-run-multus-certs\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.895985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894909    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-system-cni-dir\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.895985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894935    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-cnibin\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.895985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894959    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-kubernetes\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.895985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894963    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vvj2\" (UniqueName: \"kubernetes.io/projected/204212cc-1c4c-4bfb-8b41-03a98042166c-kube-api-access-7vvj2\") pod \"network-check-target-jh728\" (UID: \"204212cc-1c4c-4bfb-8b41-03a98042166c\") " pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:40:49.895985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894981    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-sys\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.895985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.894989    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4770276e-e9dc-4f6c-94f6-d2dfe2f6f115-tmp-dir\") pod \"node-resolver-49rdp\" (UID: \"4770276e-e9dc-4f6c-94f6-d2dfe2f6f115\") " pod="openshift-dns/node-resolver-49rdp"
Oct 30 00:40:49.895985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895016    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5fade25-18f0-4e3f-978a-5bb39a356ab0-system-cni-dir\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:40:49.895985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895029    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-var-lib-kubelet\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.895985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895042    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-multus-socket-dir-parent\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.895985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895067    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-var-lib-cni-multus\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.896630 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895088    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-modprobe-d\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.896630 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895101    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/380df728-1bf4-4258-ba04-c31e1d28a3aa-ovnkube-config\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.896630 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895110    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-lib-modules\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.896630 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895134    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f5fade25-18f0-4e3f-978a-5bb39a356ab0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:40:49.896630 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895181    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-sysctl-conf\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.896630 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895182    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f5fade25-18f0-4e3f-978a-5bb39a356ab0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:40:49.896630 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-run-systemd\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.896630 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895246    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j89wv\" (UniqueName: \"kubernetes.io/projected/380df728-1bf4-4258-ba04-c31e1d28a3aa-kube-api-access-j89wv\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.896630 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895273    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwnlf\" (UniqueName: \"kubernetes.io/projected/4770276e-e9dc-4f6c-94f6-d2dfe2f6f115-kube-api-access-lwnlf\") pod \"node-resolver-49rdp\" (UID: \"4770276e-e9dc-4f6c-94f6-d2dfe2f6f115\") " pod="openshift-dns/node-resolver-49rdp"
Oct 30 00:40:49.896630 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895301    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-log-socket\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.896630 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895326    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.896630 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895353    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/380df728-1bf4-4258-ba04-c31e1d28a3aa-env-overrides\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.896630 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895379    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-etc-openvswitch\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.896630 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895387    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4770276e-e9dc-4f6c-94f6-d2dfe2f6f115-tmp-dir\") pod \"node-resolver-49rdp\" (UID: \"4770276e-e9dc-4f6c-94f6-d2dfe2f6f115\") " pod="openshift-dns/node-resolver-49rdp"
Oct 30 00:40:49.896630 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895405    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/380df728-1bf4-4258-ba04-c31e1d28a3aa-ovnkube-script-lib\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.896630 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895449    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbef968a-7aa4-4e23-9278-3706e60d8c0b-kubelet-config\") pod \"global-pull-secret-syncer-wk65n\" (UID: \"bbef968a-7aa4-4e23-9278-3706e60d8c0b\") " pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:40:49.896630 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895486    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-socket-dir\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:40:49.897114 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895510    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-systemd\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.897114 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895534    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-slash\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.897114 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895558    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-run-ovn\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.897114 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895584    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk7mm\" (UniqueName: \"kubernetes.io/projected/f5fade25-18f0-4e3f-978a-5bb39a356ab0-kube-api-access-tk7mm\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:40:49.897114 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895612    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-hostroot\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.897114 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895638    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnf9q\" (UniqueName: \"kubernetes.io/projected/0eac4baa-d0d9-43c0-8f81-3733a026e091-kube-api-access-nnf9q\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.897114 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895661    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-systemd-units\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.897114 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895682    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-node-log\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.897114 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895704    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:40:49.897114 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895728    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5fade25-18f0-4e3f-978a-5bb39a356ab0-cni-binary-copy\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:40:49.897114 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895760    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kksvb\" (UniqueName: \"kubernetes.io/projected/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-kube-api-access-kksvb\") pod \"network-metrics-daemon-b4686\" (UID: \"0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493\") " pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:40:49.897114 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895761    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f5fade25-18f0-4e3f-978a-5bb39a356ab0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:40:49.897114 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895784    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0eac4baa-d0d9-43c0-8f81-3733a026e091-cni-binary-copy\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.897114 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895811    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-sysctl-d\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.897114 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895832    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5fade25-18f0-4e3f-978a-5bb39a356ab0-system-cni-dir\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:40:49.897114 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895836    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-run-netns\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.897114 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895877    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-run-netns\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.897934 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895879    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nkxz\" (UniqueName: \"kubernetes.io/projected/19a807d5-c007-47b4-953c-28de606ababc-kube-api-access-8nkxz\") pod \"node-ca-qdjmq\" (UID: \"19a807d5-c007-47b4-953c-28de606ababc\") " pod="openshift-image-registry/node-ca-qdjmq"
Oct 30 00:40:49.897934 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895907    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/380df728-1bf4-4258-ba04-c31e1d28a3aa-ovnkube-script-lib\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.897934 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895915    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4770276e-e9dc-4f6c-94f6-d2dfe2f6f115-hosts-file\") pod \"node-resolver-49rdp\" (UID: \"4770276e-e9dc-4f6c-94f6-d2dfe2f6f115\") " pod="openshift-dns/node-resolver-49rdp"
Oct 30 00:40:49.897934 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895955    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-run-systemd\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.897934 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895956    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ac86565-2bfb-45e4-840d-c0f3a29cdfa5-agent-certs\") pod \"konnectivity-agent-77dcb\" (UID: \"2ac86565-2bfb-45e4-840d-c0f3a29cdfa5\") " pod="kube-system/konnectivity-agent-77dcb"
Oct 30 00:40:49.897934 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.895999    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ac86565-2bfb-45e4-840d-c0f3a29cdfa5-konnectivity-ca\") pod \"konnectivity-agent-77dcb\" (UID: \"2ac86565-2bfb-45e4-840d-c0f3a29cdfa5\") " pod="kube-system/konnectivity-agent-77dcb"
Oct 30 00:40:49.897934 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.896025    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae0f3040-049e-4f4e-a6e5-206b31aeea55-tmp\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.897934 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.896036    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-system-cni-dir\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.897934 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.896050    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-cni-netd\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.897934 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.896076    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/19a807d5-c007-47b4-953c-28de606ababc-serviceca\") pod \"node-ca-qdjmq\" (UID: \"19a807d5-c007-47b4-953c-28de606ababc\") " pod="openshift-image-registry/node-ca-qdjmq"
Oct 30 00:40:49.897934 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.896103    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-run-k8s-cni-cncf-io\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.897934 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.896131    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq6ls\" (UniqueName: \"kubernetes.io/projected/c75bc9c9-74be-412b-a5ac-794f971bea31-kube-api-access-jq6ls\") pod \"iptables-alerter-ljlrc\" (UID: \"c75bc9c9-74be-412b-a5ac-794f971bea31\") " pod="openshift-network-operator/iptables-alerter-ljlrc"
Oct 30 00:40:49.897934 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.896173    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-multus-socket-dir-parent\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.897934 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.896178    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-registration-dir\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:40:49.897934 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.896230    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-registration-dir\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:40:49.897934 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.896274    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-var-lib-cni-multus\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.897934 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.896329    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-lib-modules\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.898813 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.896605    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c75bc9c9-74be-412b-a5ac-794f971bea31-iptables-alerter-script\") pod \"iptables-alerter-ljlrc\" (UID: \"c75bc9c9-74be-412b-a5ac-794f971bea31\") " pod="openshift-network-operator/iptables-alerter-ljlrc"
Oct 30 00:40:49.898813 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.896641    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-log-socket\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.898813 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.896682    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-cnibin\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.898813 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.896691    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.898813 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.896749    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbef968a-7aa4-4e23-9278-3706e60d8c0b-kubelet-config\") pod \"global-pull-secret-syncer-wk65n\" (UID: \"bbef968a-7aa4-4e23-9278-3706e60d8c0b\") " pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:40:49.898813 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.896854    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f5fade25-18f0-4e3f-978a-5bb39a356ab0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:40:49.898813 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.896856    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-socket-dir\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:40:49.898813 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.896899    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-systemd\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.898813 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.896930    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-cni-netd\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.898813 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.896943    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-slash\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.898813 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.896991    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-etc-openvswitch\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.898813 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.897033    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-run-ovn\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.898813 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.897049    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/380df728-1bf4-4258-ba04-c31e1d28a3aa-env-overrides\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.898813 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.897369    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-hostroot\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.898813 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.897566    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/19a807d5-c007-47b4-953c-28de606ababc-serviceca\") pod \"node-ca-qdjmq\" (UID: \"19a807d5-c007-47b4-953c-28de606ababc\") " pod="openshift-image-registry/node-ca-qdjmq"
Oct 30 00:40:49.898813 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.897578    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-systemd-units\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.898813 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.897628    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-node-log\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.898813 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.897631    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-run-k8s-cni-cncf-io\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.899672 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.897673    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:40:49.899672 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.897715    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-tuned\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.899672 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.897782    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5fade25-18f0-4e3f-978a-5bb39a356ab0-cnibin\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:40:49.899672 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.897901    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-modprobe-d\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.899672 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.897957    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/380df728-1bf4-4258-ba04-c31e1d28a3aa-ovn-node-metrics-cert\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.899672 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.898056    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5fade25-18f0-4e3f-978a-5bb39a356ab0-cni-binary-copy\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:40:49.899672 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.898069    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-sysctl-d\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.899672 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.898101    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4770276e-e9dc-4f6c-94f6-d2dfe2f6f115-hosts-file\") pod \"node-resolver-49rdp\" (UID: \"4770276e-e9dc-4f6c-94f6-d2dfe2f6f115\") " pod="openshift-dns/node-resolver-49rdp"
Oct 30 00:40:49.899672 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.898293    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-multus-cni-dir\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.899672 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.898363    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ac86565-2bfb-45e4-840d-c0f3a29cdfa5-konnectivity-ca\") pod \"konnectivity-agent-77dcb\" (UID: \"2ac86565-2bfb-45e4-840d-c0f3a29cdfa5\") " pod="kube-system/konnectivity-agent-77dcb"
Oct 30 00:40:49.899672 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.898443    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0eac4baa-d0d9-43c0-8f81-3733a026e091-cni-binary-copy\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.900370 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.900260    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ac86565-2bfb-45e4-840d-c0f3a29cdfa5-agent-certs\") pod \"konnectivity-agent-77dcb\" (UID: \"2ac86565-2bfb-45e4-840d-c0f3a29cdfa5\") " pod="kube-system/konnectivity-agent-77dcb"
Oct 30 00:40:49.900672 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.900651    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae0f3040-049e-4f4e-a6e5-206b31aeea55-tmp\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.927171 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.926967    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpqz6\" (UniqueName: \"kubernetes.io/projected/ae0f3040-049e-4f4e-a6e5-206b31aeea55-kube-api-access-jpqz6\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:49.928312 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:49.928284    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:40:49.928312 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:49.928314    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:40:49.928510 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:49.928330    2582 projected.go:196] Error preparing data for projected volume kube-api-access-7vvj2 for pod openshift-network-diagnostics/network-check-target-jh728: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:40:49.928510 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:49.928399    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/204212cc-1c4c-4bfb-8b41-03a98042166c-kube-api-access-7vvj2 podName:204212cc-1c4c-4bfb-8b41-03a98042166c nodeName:}" failed. No retries permitted until 2026-10-30 00:40:50.428378755 +0000 UTC m=+3.120639029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7vvj2" (UniqueName: "kubernetes.io/projected/204212cc-1c4c-4bfb-8b41-03a98042166c-kube-api-access-7vvj2") pod "network-check-target-jh728" (UID: "204212cc-1c4c-4bfb-8b41-03a98042166c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:40:49.928694 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.928669    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kksvb\" (UniqueName: \"kubernetes.io/projected/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-kube-api-access-kksvb\") pod \"network-metrics-daemon-b4686\" (UID: \"0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493\") " pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:40:49.930946 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.930891    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq6ls\" (UniqueName: \"kubernetes.io/projected/c75bc9c9-74be-412b-a5ac-794f971bea31-kube-api-access-jq6ls\") pod \"iptables-alerter-ljlrc\" (UID: \"c75bc9c9-74be-412b-a5ac-794f971bea31\") " pod="openshift-network-operator/iptables-alerter-ljlrc"
Oct 30 00:40:49.931745 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.931594    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nkxz\" (UniqueName: \"kubernetes.io/projected/19a807d5-c007-47b4-953c-28de606ababc-kube-api-access-8nkxz\") pod \"node-ca-qdjmq\" (UID: \"19a807d5-c007-47b4-953c-28de606ababc\") " pod="openshift-image-registry/node-ca-qdjmq"
Oct 30 00:40:49.931745 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.931619    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk7mm\" (UniqueName: \"kubernetes.io/projected/f5fade25-18f0-4e3f-978a-5bb39a356ab0-kube-api-access-tk7mm\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:40:49.932237 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.932058    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwnlf\" (UniqueName: \"kubernetes.io/projected/4770276e-e9dc-4f6c-94f6-d2dfe2f6f115-kube-api-access-lwnlf\") pod \"node-resolver-49rdp\" (UID: \"4770276e-e9dc-4f6c-94f6-d2dfe2f6f115\") " pod="openshift-dns/node-resolver-49rdp"
Oct 30 00:40:49.932655 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.932633    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnf9q\" (UniqueName: \"kubernetes.io/projected/0eac4baa-d0d9-43c0-8f81-3733a026e091-kube-api-access-nnf9q\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:40:49.933028 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.932950    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j89wv\" (UniqueName: \"kubernetes.io/projected/380df728-1bf4-4258-ba04-c31e1d28a3aa-kube-api-access-j89wv\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:49.941508 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:49.941474    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpn92\" (UniqueName: \"kubernetes.io/projected/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-kube-api-access-bpn92\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:40:50.028649 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.028601    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:40:50.040791 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:50.040751    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae0f3040_049e_4f4e_a6e5_206b31aeea55.slice/crio-330d2a28006f1438dd2980a0c252a5e8a535cc933d4cfe3f3b660c389af56c99 WatchSource:0}: Error finding container 330d2a28006f1438dd2980a0c252a5e8a535cc933d4cfe3f3b660c389af56c99: Status 404 returned error can't find the container with id 330d2a28006f1438dd2980a0c252a5e8a535cc933d4cfe3f3b660c389af56c99
Oct 30 00:40:50.061521 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.061471    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ljlrc"
Oct 30 00:40:50.065840 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.065817    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-77dcb"
Oct 30 00:40:50.068435 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.068418    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:40:50.071114 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.071071    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qp98l"
Oct 30 00:40:50.074570 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:50.074544    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75bc9c9_74be_412b_a5ac_794f971bea31.slice/crio-666786686b632a96170c0208d19941a722053a47b1c493774fcf7227ea91797c WatchSource:0}: Error finding container 666786686b632a96170c0208d19941a722053a47b1c493774fcf7227ea91797c: Status 404 returned error can't find the container with id 666786686b632a96170c0208d19941a722053a47b1c493774fcf7227ea91797c
Oct 30 00:40:50.080928 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.080907    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qdjmq"
Oct 30 00:40:50.082693 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:50.082668    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac86565_2bfb_45e4_840d_c0f3a29cdfa5.slice/crio-e4346845caaf9b198310d004c52c7d155f38f4caf38a4b1079ff4739a59b3406 WatchSource:0}: Error finding container e4346845caaf9b198310d004c52c7d155f38f4caf38a4b1079ff4739a59b3406: Status 404 returned error can't find the container with id e4346845caaf9b198310d004c52c7d155f38f4caf38a4b1079ff4739a59b3406
Oct 30 00:40:50.083949 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:50.083897    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380df728_1bf4_4258_ba04_c31e1d28a3aa.slice/crio-f14f1ab09e47b335bc8ab3c39be0ac2a6416c96143fec1488db1a8266128472d WatchSource:0}: Error finding container f14f1ab09e47b335bc8ab3c39be0ac2a6416c96143fec1488db1a8266128472d: Status 404 returned error can't find the container with id f14f1ab09e47b335bc8ab3c39be0ac2a6416c96143fec1488db1a8266128472d
Oct 30 00:40:50.092723 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.092699    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:40:50.093302 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:50.093250    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a807d5_c007_47b4_953c_28de606ababc.slice/crio-4a387f3dc74cf8bf7a52fa606628b8212707b6a2bc64826138ea18bb1e2b93b6 WatchSource:0}: Error finding container 4a387f3dc74cf8bf7a52fa606628b8212707b6a2bc64826138ea18bb1e2b93b6: Status 404 returned error can't find the container with id 4a387f3dc74cf8bf7a52fa606628b8212707b6a2bc64826138ea18bb1e2b93b6
Oct 30 00:40:50.104423 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.104390    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-49rdp"
Oct 30 00:40:50.105163 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:50.105140    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d77e73_a34c_4df8_b1fc_d8e3c1d2e074.slice/crio-2b3fba6489f614f481bcba72e9e08c34e049a7410f507c517a286b12792bcf13 WatchSource:0}: Error finding container 2b3fba6489f614f481bcba72e9e08c34e049a7410f507c517a286b12792bcf13: Status 404 returned error can't find the container with id 2b3fba6489f614f481bcba72e9e08c34e049a7410f507c517a286b12792bcf13
Oct 30 00:40:50.111722 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.111696    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:40:50.118244 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:50.118217    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4770276e_e9dc_4f6c_94f6_d2dfe2f6f115.slice/crio-1aa45a060b9b9d4d37b976fdbc291e098d6c2fb0cde1b742e32f4c411b49b01c WatchSource:0}: Error finding container 1aa45a060b9b9d4d37b976fdbc291e098d6c2fb0cde1b742e32f4c411b49b01c: Status 404 returned error can't find the container with id 1aa45a060b9b9d4d37b976fdbc291e098d6c2fb0cde1b742e32f4c411b49b01c
Oct 30 00:40:50.124431 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:40:50.124404    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5fade25_18f0_4e3f_978a_5bb39a356ab0.slice/crio-1940788f530ef326fd10fac6b6036bada7821b1156f1720b6ab940debdea870d WatchSource:0}: Error finding container 1940788f530ef326fd10fac6b6036bada7821b1156f1720b6ab940debdea870d: Status 404 returned error can't find the container with id 1940788f530ef326fd10fac6b6036bada7821b1156f1720b6ab940debdea870d
Oct 30 00:40:50.400308 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.399991    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbef968a-7aa4-4e23-9278-3706e60d8c0b-original-pull-secret\") pod \"global-pull-secret-syncer-wk65n\" (UID: \"bbef968a-7aa4-4e23-9278-3706e60d8c0b\") " pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:40:50.400308 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.400105    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs\") pod \"network-metrics-daemon-b4686\" (UID: \"0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493\") " pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:40:50.400308 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:50.400167    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:40:50.400308 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:50.400228    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:40:50.400308 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:50.400236    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbef968a-7aa4-4e23-9278-3706e60d8c0b-original-pull-secret podName:bbef968a-7aa4-4e23-9278-3706e60d8c0b nodeName:}" failed. No retries permitted until 2026-10-30 00:40:51.400219252 +0000 UTC m=+4.092479527 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbef968a-7aa4-4e23-9278-3706e60d8c0b-original-pull-secret") pod "global-pull-secret-syncer-wk65n" (UID: "bbef968a-7aa4-4e23-9278-3706e60d8c0b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:40:50.400308 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:50.400282    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs podName:0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493 nodeName:}" failed. No retries permitted until 2026-10-30 00:40:51.400264928 +0000 UTC m=+4.092525192 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs") pod "network-metrics-daemon-b4686" (UID: "0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:40:50.500621 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.500591    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vvj2\" (UniqueName: \"kubernetes.io/projected/204212cc-1c4c-4bfb-8b41-03a98042166c-kube-api-access-7vvj2\") pod \"network-check-target-jh728\" (UID: \"204212cc-1c4c-4bfb-8b41-03a98042166c\") " pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:40:50.500851 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:50.500831    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:40:50.501000 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:50.500856    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:40:50.501000 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:50.500870    2582 projected.go:196] Error preparing data for projected volume kube-api-access-7vvj2 for pod openshift-network-diagnostics/network-check-target-jh728: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:40:50.501160 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:50.501011    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/204212cc-1c4c-4bfb-8b41-03a98042166c-kube-api-access-7vvj2 podName:204212cc-1c4c-4bfb-8b41-03a98042166c nodeName:}" failed. No retries permitted until 2026-10-30 00:40:51.500988445 +0000 UTC m=+4.193248721 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7vvj2" (UniqueName: "kubernetes.io/projected/204212cc-1c4c-4bfb-8b41-03a98042166c-kube-api-access-7vvj2") pod "network-check-target-jh728" (UID: "204212cc-1c4c-4bfb-8b41-03a98042166c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:40:50.777740 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.777478    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:35:50 +0000 UTC" deadline="2028-06-15 00:33:19.689696737 +0000 UTC"
Oct 30 00:40:50.777740 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.777550    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14255h52m28.91215127s"
Oct 30 00:40:50.839894 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.834906    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:40:50.839894 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:50.835337    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh728" podUID="204212cc-1c4c-4bfb-8b41-03a98042166c"
Oct 30 00:40:50.850434 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.850350    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ljlrc" event={"ID":"c75bc9c9-74be-412b-a5ac-794f971bea31","Type":"ContainerStarted","Data":"666786686b632a96170c0208d19941a722053a47b1c493774fcf7227ea91797c"}
Oct 30 00:40:50.862847 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.862809    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-210.ec2.internal" event={"ID":"17edaf9736d7ee1e0409a6d573ccb434","Type":"ContainerStarted","Data":"6f7a116041a25cf213d0e932612dd9d753549f37fbc586e07552b75c98efb608"}
Oct 30 00:40:50.872916 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.872631    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn" event={"ID":"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074","Type":"ContainerStarted","Data":"2b3fba6489f614f481bcba72e9e08c34e049a7410f507c517a286b12792bcf13"}
Oct 30 00:40:50.879024 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.876190    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qp98l" event={"ID":"0eac4baa-d0d9-43c0-8f81-3733a026e091","Type":"ContainerStarted","Data":"5ebac1c448b6d1cd408bc250e18f054eb6714965646f9d0411ca4c508f4ec837"}
Oct 30 00:40:50.885989 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.885882    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7855k" event={"ID":"380df728-1bf4-4258-ba04-c31e1d28a3aa","Type":"ContainerStarted","Data":"f14f1ab09e47b335bc8ab3c39be0ac2a6416c96143fec1488db1a8266128472d"}
Oct 30 00:40:50.898231 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.898129    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q4bch" event={"ID":"ae0f3040-049e-4f4e-a6e5-206b31aeea55","Type":"ContainerStarted","Data":"330d2a28006f1438dd2980a0c252a5e8a535cc933d4cfe3f3b660c389af56c99"}
Oct 30 00:40:50.911100 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.911043    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-210.ec2.internal" podStartSLOduration=1.911025757 podStartE2EDuration="1.911025757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:40:50.910821325 +0000 UTC m=+3.603081610" watchObservedRunningTime="2026-10-30 00:40:50.911025757 +0000 UTC m=+3.603286041"
Oct 30 00:40:50.918118 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.917290    2582 generic.go:356] "Generic (PLEG): container finished" podID="e4c9778b041df41465cbcba4cba59e17" containerID="78b07aba50b7255e46ab7007879cec55a1150e1b3d4a74927949cc6efb7ead09" exitCode=0
Oct 30 00:40:50.918118 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.917375    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal" event={"ID":"e4c9778b041df41465cbcba4cba59e17","Type":"ContainerDied","Data":"78b07aba50b7255e46ab7007879cec55a1150e1b3d4a74927949cc6efb7ead09"}
Oct 30 00:40:50.936369 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.935361    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tz5vn" event={"ID":"f5fade25-18f0-4e3f-978a-5bb39a356ab0","Type":"ContainerStarted","Data":"1940788f530ef326fd10fac6b6036bada7821b1156f1720b6ab940debdea870d"}
Oct 30 00:40:50.960714 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.960583    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-49rdp" event={"ID":"4770276e-e9dc-4f6c-94f6-d2dfe2f6f115","Type":"ContainerStarted","Data":"1aa45a060b9b9d4d37b976fdbc291e098d6c2fb0cde1b742e32f4c411b49b01c"}
Oct 30 00:40:50.965524 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.965439    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qdjmq" event={"ID":"19a807d5-c007-47b4-953c-28de606ababc","Type":"ContainerStarted","Data":"4a387f3dc74cf8bf7a52fa606628b8212707b6a2bc64826138ea18bb1e2b93b6"}
Oct 30 00:40:50.971010 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:50.970979    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-77dcb" event={"ID":"2ac86565-2bfb-45e4-840d-c0f3a29cdfa5","Type":"ContainerStarted","Data":"e4346845caaf9b198310d004c52c7d155f38f4caf38a4b1079ff4739a59b3406"}
Oct 30 00:40:51.408973 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:51.407761    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbef968a-7aa4-4e23-9278-3706e60d8c0b-original-pull-secret\") pod \"global-pull-secret-syncer-wk65n\" (UID: \"bbef968a-7aa4-4e23-9278-3706e60d8c0b\") " pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:40:51.408973 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:51.407894    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs\") pod \"network-metrics-daemon-b4686\" (UID: \"0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493\") " pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:40:51.408973 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:51.408026    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:40:51.408973 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:51.408092    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs podName:0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493 nodeName:}" failed. No retries permitted until 2026-10-30 00:40:53.40807365 +0000 UTC m=+6.100333912 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs") pod "network-metrics-daemon-b4686" (UID: "0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:40:51.408973 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:51.408544    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:40:51.408973 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:51.408597    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbef968a-7aa4-4e23-9278-3706e60d8c0b-original-pull-secret podName:bbef968a-7aa4-4e23-9278-3706e60d8c0b nodeName:}" failed. No retries permitted until 2026-10-30 00:40:53.408582336 +0000 UTC m=+6.100842612 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbef968a-7aa4-4e23-9278-3706e60d8c0b-original-pull-secret") pod "global-pull-secret-syncer-wk65n" (UID: "bbef968a-7aa4-4e23-9278-3706e60d8c0b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:40:51.508390 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:51.508345    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vvj2\" (UniqueName: \"kubernetes.io/projected/204212cc-1c4c-4bfb-8b41-03a98042166c-kube-api-access-7vvj2\") pod \"network-check-target-jh728\" (UID: \"204212cc-1c4c-4bfb-8b41-03a98042166c\") " pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:40:51.508562 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:51.508530    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:40:51.508562 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:51.508551    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:40:51.508683 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:51.508565    2582 projected.go:196] Error preparing data for projected volume kube-api-access-7vvj2 for pod openshift-network-diagnostics/network-check-target-jh728: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:40:51.508683 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:51.508623    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/204212cc-1c4c-4bfb-8b41-03a98042166c-kube-api-access-7vvj2 podName:204212cc-1c4c-4bfb-8b41-03a98042166c nodeName:}" failed. No retries permitted until 2026-10-30 00:40:53.508604344 +0000 UTC m=+6.200864619 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7vvj2" (UniqueName: "kubernetes.io/projected/204212cc-1c4c-4bfb-8b41-03a98042166c-kube-api-access-7vvj2") pod "network-check-target-jh728" (UID: "204212cc-1c4c-4bfb-8b41-03a98042166c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:40:51.778319 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:51.778243    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:35:50 +0000 UTC" deadline="2028-06-12 14:50:44.998831914 +0000 UTC"
Oct 30 00:40:51.778319 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:51.778283    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14198h9m53.220554009s"
Oct 30 00:40:51.842168 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:51.837864    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:40:51.842168 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:51.838010    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b4686" podUID="0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493"
Oct 30 00:40:51.842168 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:51.838228    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:40:51.842168 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:51.838620    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wk65n" podUID="bbef968a-7aa4-4e23-9278-3706e60d8c0b"
Oct 30 00:40:52.008809 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:52.008767    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal" event={"ID":"e4c9778b041df41465cbcba4cba59e17","Type":"ContainerStarted","Data":"a16c7065be591119eaac96e1e8818460066d2771465d4fd346e7a0d702cc8912"}
Oct 30 00:40:52.834898 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:52.834819    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:40:52.835384 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:52.834955    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh728" podUID="204212cc-1c4c-4bfb-8b41-03a98042166c"
Oct 30 00:40:53.054169 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:53.038742    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tz5vn" event={"ID":"f5fade25-18f0-4e3f-978a-5bb39a356ab0","Type":"ContainerStarted","Data":"b74eb6e2959251917f1841053f8acb77e46517f2743e58af83e07a9f6a7423e3"}
Oct 30 00:40:53.096699 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:53.096599    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal" podStartSLOduration=4.09657707 podStartE2EDuration="4.09657707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:40:52.06476448 +0000 UTC m=+4.757024766" watchObservedRunningTime="2026-10-30 00:40:53.09657707 +0000 UTC m=+5.788837356"
Oct 30 00:40:53.430961 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:53.430884    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs\") pod \"network-metrics-daemon-b4686\" (UID: \"0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493\") " pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:40:53.430961 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:53.430936    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbef968a-7aa4-4e23-9278-3706e60d8c0b-original-pull-secret\") pod \"global-pull-secret-syncer-wk65n\" (UID: \"bbef968a-7aa4-4e23-9278-3706e60d8c0b\") " pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:40:53.431179 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:53.431058    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:40:53.431179 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:53.431133    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbef968a-7aa4-4e23-9278-3706e60d8c0b-original-pull-secret podName:bbef968a-7aa4-4e23-9278-3706e60d8c0b nodeName:}" failed. No retries permitted until 2026-10-30 00:40:57.43111411 +0000 UTC m=+10.123374386 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbef968a-7aa4-4e23-9278-3706e60d8c0b-original-pull-secret") pod "global-pull-secret-syncer-wk65n" (UID: "bbef968a-7aa4-4e23-9278-3706e60d8c0b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:40:53.431531 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:53.431510    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:40:53.431619 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:53.431576    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs podName:0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493 nodeName:}" failed. No retries permitted until 2026-10-30 00:40:57.431557437 +0000 UTC m=+10.123817708 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs") pod "network-metrics-daemon-b4686" (UID: "0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:40:53.532709 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:53.531887    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vvj2\" (UniqueName: \"kubernetes.io/projected/204212cc-1c4c-4bfb-8b41-03a98042166c-kube-api-access-7vvj2\") pod \"network-check-target-jh728\" (UID: \"204212cc-1c4c-4bfb-8b41-03a98042166c\") " pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:40:53.532709 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:53.532128    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:40:53.532709 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:53.532149    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:40:53.532709 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:53.532162    2582 projected.go:196] Error preparing data for projected volume kube-api-access-7vvj2 for pod openshift-network-diagnostics/network-check-target-jh728: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:40:53.532709 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:53.532214    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/204212cc-1c4c-4bfb-8b41-03a98042166c-kube-api-access-7vvj2 podName:204212cc-1c4c-4bfb-8b41-03a98042166c nodeName:}" failed. No retries permitted until 2026-10-30 00:40:57.532197778 +0000 UTC m=+10.224458039 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7vvj2" (UniqueName: "kubernetes.io/projected/204212cc-1c4c-4bfb-8b41-03a98042166c-kube-api-access-7vvj2") pod "network-check-target-jh728" (UID: "204212cc-1c4c-4bfb-8b41-03a98042166c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:40:53.835081 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:53.835036    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:40:53.835558 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:53.835191    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b4686" podUID="0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493"
Oct 30 00:40:53.837836 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:53.837805    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:40:53.837963 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:53.837918    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wk65n" podUID="bbef968a-7aa4-4e23-9278-3706e60d8c0b"
Oct 30 00:40:54.045425 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:54.045390    2582 generic.go:356] "Generic (PLEG): container finished" podID="f5fade25-18f0-4e3f-978a-5bb39a356ab0" containerID="b74eb6e2959251917f1841053f8acb77e46517f2743e58af83e07a9f6a7423e3" exitCode=0
Oct 30 00:40:54.045540 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:54.045442    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tz5vn" event={"ID":"f5fade25-18f0-4e3f-978a-5bb39a356ab0","Type":"ContainerDied","Data":"b74eb6e2959251917f1841053f8acb77e46517f2743e58af83e07a9f6a7423e3"}
Oct 30 00:40:54.834752 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:54.834719    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:40:54.834918 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:54.834843    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh728" podUID="204212cc-1c4c-4bfb-8b41-03a98042166c"
Oct 30 00:40:55.072896 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:55.072210    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-77dcb" event={"ID":"2ac86565-2bfb-45e4-840d-c0f3a29cdfa5","Type":"ContainerStarted","Data":"2cc72acbb4b0b7eac9c7723a8a5eccc08321b664341765b24721eb9b5fd7be73"}
Oct 30 00:40:55.116880 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:55.115952    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-77dcb" podStartSLOduration=3.177442919 podStartE2EDuration="7.11593328s" totalImagesPullingTime="3.938490361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:40:55.115572134 +0000 UTC m=+7.807832429" watchObservedRunningTime="2026-10-30 00:40:55.11593328 +0000 UTC m=+7.808193559"
Oct 30 00:40:55.836264 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:55.836194    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:40:55.836435 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:55.836337    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wk65n" podUID="bbef968a-7aa4-4e23-9278-3706e60d8c0b"
Oct 30 00:40:55.836701 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:55.836528    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:40:55.836701 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:55.836657    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b4686" podUID="0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493"
Oct 30 00:40:56.080473 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:56.080196    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qdjmq" event={"ID":"19a807d5-c007-47b4-953c-28de606ababc","Type":"ContainerStarted","Data":"13be70fd18c995016fb9b8a14717c4f32e4967608ee0af1e4c85ba4a780ddfcf"}
Oct 30 00:40:56.835778 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:56.835315    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:40:56.835778 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:56.835438    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh728" podUID="204212cc-1c4c-4bfb-8b41-03a98042166c"
Oct 30 00:40:57.085104 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:57.085059    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn" event={"ID":"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074","Type":"ContainerStarted","Data":"34464cfe7ffc818ea8d2185fb3409f9f489ec5e0b004573e16717a25f4a9ecbd"}
Oct 30 00:40:57.467062 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:57.467023    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs\") pod \"network-metrics-daemon-b4686\" (UID: \"0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493\") " pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:40:57.467367 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:57.467315    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:40:57.467474 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:57.467426    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs podName:0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:05.467401452 +0000 UTC m=+18.159661709 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs") pod "network-metrics-daemon-b4686" (UID: "0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:40:57.467968 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:57.467942    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbef968a-7aa4-4e23-9278-3706e60d8c0b-original-pull-secret\") pod \"global-pull-secret-syncer-wk65n\" (UID: \"bbef968a-7aa4-4e23-9278-3706e60d8c0b\") " pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:40:57.468094 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:57.468065    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:40:57.468166 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:57.468136    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbef968a-7aa4-4e23-9278-3706e60d8c0b-original-pull-secret podName:bbef968a-7aa4-4e23-9278-3706e60d8c0b nodeName:}" failed. No retries permitted until 2026-10-30 00:41:05.468120088 +0000 UTC m=+18.160380356 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbef968a-7aa4-4e23-9278-3706e60d8c0b-original-pull-secret") pod "global-pull-secret-syncer-wk65n" (UID: "bbef968a-7aa4-4e23-9278-3706e60d8c0b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:40:57.568856 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:57.568829    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vvj2\" (UniqueName: \"kubernetes.io/projected/204212cc-1c4c-4bfb-8b41-03a98042166c-kube-api-access-7vvj2\") pod \"network-check-target-jh728\" (UID: \"204212cc-1c4c-4bfb-8b41-03a98042166c\") " pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:40:57.569024 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:57.568993    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:40:57.569024 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:57.569016    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:40:57.569159 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:57.569028    2582 projected.go:196] Error preparing data for projected volume kube-api-access-7vvj2 for pod openshift-network-diagnostics/network-check-target-jh728: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:40:57.569159 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:57.569153    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/204212cc-1c4c-4bfb-8b41-03a98042166c-kube-api-access-7vvj2 podName:204212cc-1c4c-4bfb-8b41-03a98042166c nodeName:}" failed. No retries permitted until 2026-10-30 00:41:05.569061182 +0000 UTC m=+18.261321443 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7vvj2" (UniqueName: "kubernetes.io/projected/204212cc-1c4c-4bfb-8b41-03a98042166c-kube-api-access-7vvj2") pod "network-check-target-jh728" (UID: "204212cc-1c4c-4bfb-8b41-03a98042166c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:40:57.612210 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:57.612185    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-77dcb"
Oct 30 00:40:57.612863 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:57.612797    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-77dcb"
Oct 30 00:40:57.671212 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:57.671160    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qdjmq" podStartSLOduration=4.846257759 podStartE2EDuration="9.67114209s" totalImagesPullingTime="4.824884331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:40:56.123192951 +0000 UTC m=+8.815453231" watchObservedRunningTime="2026-10-30 00:40:57.67114209 +0000 UTC m=+10.363402370"
Oct 30 00:40:57.724248 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:57.724193    2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 00:40:57.753804 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:57.753128    2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:40:57.724214156Z","UUID":"865da3b5-31b2-4b9d-9838-d3ab5e6eb48f","Handler":null,"Name":"","Endpoint":""}
Oct 30 00:40:57.757045 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:57.755658    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 00:40:57.757045 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:57.755684    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 00:40:57.836544 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:57.836388    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:40:57.836544 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:57.836491    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wk65n" podUID="bbef968a-7aa4-4e23-9278-3706e60d8c0b"
Oct 30 00:40:57.836941 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:57.836797    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:40:57.836941 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:57.836903    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b4686" podUID="0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493"
Oct 30 00:40:58.089935 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:58.089901    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn" event={"ID":"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074","Type":"ContainerStarted","Data":"5929403faa0334a08d21d66e128703ce3966e6715ccef67b54ffb1ea14ccb241"}
Oct 30 00:40:58.092924 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:58.092706    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qp98l" event={"ID":"0eac4baa-d0d9-43c0-8f81-3733a026e091","Type":"ContainerStarted","Data":"80c707cb3b105aa2e7f0ffdfed9fc586965d7b22e4b841ca8b37cdf7e2808506"}
Oct 30 00:40:58.096739 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:58.096323    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-49rdp" event={"ID":"4770276e-e9dc-4f6c-94f6-d2dfe2f6f115","Type":"ContainerStarted","Data":"0d271e58ff7c8908735a0d6f301544ec39d86f8b49c4552ee555367fd94308b8"}
Oct 30 00:40:58.221666 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:58.220539    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-qp98l" podStartSLOduration=2.409850981 podStartE2EDuration="10.220520029s" totalImagesPullingTime="7.810669048s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:40:58.140684436 +0000 UTC m=+10.832944717" watchObservedRunningTime="2026-10-30 00:40:58.220520029 +0000 UTC m=+10.912780309"
Oct 30 00:40:58.221666 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:58.221363    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-49rdp" podStartSLOduration=2.525046435 podStartE2EDuration="10.221352067s" totalImagesPullingTime="7.696305632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:40:58.220291714 +0000 UTC m=+10.912552006" watchObservedRunningTime="2026-10-30 00:40:58.221352067 +0000 UTC m=+10.913612352"
Oct 30 00:40:58.834933 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:58.834757    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:40:58.835065 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:58.835036    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh728" podUID="204212cc-1c4c-4bfb-8b41-03a98042166c"
Oct 30 00:40:59.100892 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:59.100792    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn" event={"ID":"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074","Type":"ContainerStarted","Data":"433230176e57c5cce0b936bc0f2891c6af3050edc51b1f009b60db1ad4fe83cc"}
Oct 30 00:40:59.103933 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:59.103905    2582 generic.go:356] "Generic (PLEG): container finished" podID="f5fade25-18f0-4e3f-978a-5bb39a356ab0" containerID="8f4fcb280a80b28a33e6695b6feb61747757f015f0a45207ad16b0b5fcb541b7" exitCode=0
Oct 30 00:40:59.104086 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:59.104016    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tz5vn" event={"ID":"f5fade25-18f0-4e3f-978a-5bb39a356ab0","Type":"ContainerDied","Data":"8f4fcb280a80b28a33e6695b6feb61747757f015f0a45207ad16b0b5fcb541b7"}
Oct 30 00:40:59.152583 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:59.152523    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn" podStartSLOduration=2.997168786 podStartE2EDuration="11.152504424s" totalImagesPullingTime="8.155335638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:40:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:40:59.152099463 +0000 UTC m=+11.844359739" watchObservedRunningTime="2026-10-30 00:40:59.152504424 +0000 UTC m=+11.844764707"
Oct 30 00:40:59.835114 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:59.835064    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:40:59.835244 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:59.835202    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b4686" podUID="0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493"
Oct 30 00:40:59.835348 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:40:59.835252    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:40:59.835348 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:40:59.835340    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wk65n" podUID="bbef968a-7aa4-4e23-9278-3706e60d8c0b"
Oct 30 00:41:00.108087 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:00.108045    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ljlrc" event={"ID":"c75bc9c9-74be-412b-a5ac-794f971bea31","Type":"ContainerStarted","Data":"d950e148ffcbc7035360451120be391870e2e6beb2e81d18afcba7d77a83a997"}
Oct 30 00:41:00.151026 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:00.150978    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ljlrc" podStartSLOduration=3.970592439 podStartE2EDuration="12.150966113s" totalImagesPullingTime="8.180373674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:00.150845758 +0000 UTC m=+12.843106038" watchObservedRunningTime="2026-10-30 00:41:00.150966113 +0000 UTC m=+12.843226392"
Oct 30 00:41:00.835409 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:00.835382    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:41:00.835596 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:00.835480    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh728" podUID="204212cc-1c4c-4bfb-8b41-03a98042166c"
Oct 30 00:41:01.111719 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:01.111632    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q4bch" event={"ID":"ae0f3040-049e-4f4e-a6e5-206b31aeea55","Type":"ContainerStarted","Data":"64eff9808d6601177da72a74eb94d7b842846a48511cef6b5b6f54489b23d827"}
Oct 30 00:41:01.114129 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:01.114103    2582 generic.go:356] "Generic (PLEG): container finished" podID="f5fade25-18f0-4e3f-978a-5bb39a356ab0" containerID="01a246586e0cc9a138277d595d7a76b254390f6c1d7f06c71e6d177180e6fa8e" exitCode=0
Oct 30 00:41:01.114274 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:01.114181    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tz5vn" event={"ID":"f5fade25-18f0-4e3f-978a-5bb39a356ab0","Type":"ContainerDied","Data":"01a246586e0cc9a138277d595d7a76b254390f6c1d7f06c71e6d177180e6fa8e"}
Oct 30 00:41:01.157040 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:01.156988    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q4bch" podStartSLOduration=2.7640328739999998 podStartE2EDuration="13.156975117s" totalImagesPullingTime="10.392942243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:01.156761664 +0000 UTC m=+13.849021944" watchObservedRunningTime="2026-10-30 00:41:01.156975117 +0000 UTC m=+13.849235395"
Oct 30 00:41:01.835447 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:01.835417    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:41:01.835606 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:01.835542    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wk65n" podUID="bbef968a-7aa4-4e23-9278-3706e60d8c0b"
Oct 30 00:41:01.835606 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:01.835591    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:41:01.835725 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:01.835704    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b4686" podUID="0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493"
Oct 30 00:41:02.122350 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:02.122258    2582 generic.go:356] "Generic (PLEG): container finished" podID="f5fade25-18f0-4e3f-978a-5bb39a356ab0" containerID="9519e32db0c84191cff88665ef3b933d8e0907e7bb53786d97768848c301eb9f" exitCode=0
Oct 30 00:41:02.123011 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:02.122347    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tz5vn" event={"ID":"f5fade25-18f0-4e3f-978a-5bb39a356ab0","Type":"ContainerDied","Data":"9519e32db0c84191cff88665ef3b933d8e0907e7bb53786d97768848c301eb9f"}
Oct 30 00:41:02.835026 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:02.834992    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:41:02.835201 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:02.835137    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh728" podUID="204212cc-1c4c-4bfb-8b41-03a98042166c"
Oct 30 00:41:03.836827 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:03.836616    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:41:03.837269 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:03.836919    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wk65n" podUID="bbef968a-7aa4-4e23-9278-3706e60d8c0b"
Oct 30 00:41:03.837338 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:03.837279    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:41:03.837387 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:03.837370    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b4686" podUID="0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493"
Oct 30 00:41:04.128386 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:04.128296    2582 generic.go:356] "Generic (PLEG): container finished" podID="f5fade25-18f0-4e3f-978a-5bb39a356ab0" containerID="34700786c0ee9bd743338750d8da2f7d262537fd94d2f53b12ca49d79c96c06d" exitCode=0
Oct 30 00:41:04.128386 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:04.128334    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tz5vn" event={"ID":"f5fade25-18f0-4e3f-978a-5bb39a356ab0","Type":"ContainerDied","Data":"34700786c0ee9bd743338750d8da2f7d262537fd94d2f53b12ca49d79c96c06d"}
Oct 30 00:41:04.835432 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:04.835409    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:41:04.835549 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:04.835528    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh728" podUID="204212cc-1c4c-4bfb-8b41-03a98042166c"
Oct 30 00:41:05.134195 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:05.133982    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7855k" event={"ID":"380df728-1bf4-4258-ba04-c31e1d28a3aa","Type":"ContainerStarted","Data":"29622801e9802b883da3093d199f6d52e681bf1c2d1dd6dbb219123103d4bf4e"}
Oct 30 00:41:05.134628 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:05.134336    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7855k" event={"ID":"380df728-1bf4-4258-ba04-c31e1d28a3aa","Type":"ContainerStarted","Data":"2ca0a96acac8ddd20efb899030a06947568200a287193283c2e58887f23ca8ac"}
Oct 30 00:41:05.134628 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:05.134394    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7855k" event={"ID":"380df728-1bf4-4258-ba04-c31e1d28a3aa","Type":"ContainerStarted","Data":"2cbe1bc3bd57f45f01cda04d2753dddc346f6aa243de6e50f05ebc0ae9f6a3cd"}
Oct 30 00:41:05.134628 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:05.134443    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7855k" event={"ID":"380df728-1bf4-4258-ba04-c31e1d28a3aa","Type":"ContainerStarted","Data":"8a927472edef6279af8d4850ea1920edae56f368629d815cdf588e955bdf3082"}
Oct 30 00:41:05.134628 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:05.134491    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7855k" event={"ID":"380df728-1bf4-4258-ba04-c31e1d28a3aa","Type":"ContainerStarted","Data":"b5a6ee0272d65735fc9686a7f111475e37668b34e5bd9c99d6be66b0361ca4a1"}
Oct 30 00:41:05.134628 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:05.134536    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7855k" event={"ID":"380df728-1bf4-4258-ba04-c31e1d28a3aa","Type":"ContainerStarted","Data":"6a720ed1eb5d73e6dff2972c92e7f271f22a95c00c4aff3cc6824b8498751b64"}
Oct 30 00:41:05.137063 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:05.137038    2582 generic.go:356] "Generic (PLEG): container finished" podID="f5fade25-18f0-4e3f-978a-5bb39a356ab0" containerID="abb929bbd3d85b3408431b1db469efaae4b06a19c4e58a22651f45ac5ee053b4" exitCode=0
Oct 30 00:41:05.137194 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:05.137089    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tz5vn" event={"ID":"f5fade25-18f0-4e3f-978a-5bb39a356ab0","Type":"ContainerDied","Data":"abb929bbd3d85b3408431b1db469efaae4b06a19c4e58a22651f45ac5ee053b4"}
Oct 30 00:41:05.533136 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:05.533105    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs\") pod \"network-metrics-daemon-b4686\" (UID: \"0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493\") " pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:41:05.533136 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:05.533140    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbef968a-7aa4-4e23-9278-3706e60d8c0b-original-pull-secret\") pod \"global-pull-secret-syncer-wk65n\" (UID: \"bbef968a-7aa4-4e23-9278-3706e60d8c0b\") " pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:41:05.533342 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:05.533236    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:41:05.533342 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:05.533239    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:41:05.533342 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:05.533285    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbef968a-7aa4-4e23-9278-3706e60d8c0b-original-pull-secret podName:bbef968a-7aa4-4e23-9278-3706e60d8c0b nodeName:}" failed. No retries permitted until 2026-10-30 00:41:21.53327198 +0000 UTC m=+34.225532237 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbef968a-7aa4-4e23-9278-3706e60d8c0b-original-pull-secret") pod "global-pull-secret-syncer-wk65n" (UID: "bbef968a-7aa4-4e23-9278-3706e60d8c0b") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:41:05.533342 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:05.533297    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs podName:0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:21.533291611 +0000 UTC m=+34.225551868 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs") pod "network-metrics-daemon-b4686" (UID: "0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:41:05.634297 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:05.634264    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vvj2\" (UniqueName: \"kubernetes.io/projected/204212cc-1c4c-4bfb-8b41-03a98042166c-kube-api-access-7vvj2\") pod \"network-check-target-jh728\" (UID: \"204212cc-1c4c-4bfb-8b41-03a98042166c\") " pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:41:05.634461 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:05.634389    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:41:05.634461 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:05.634403    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:41:05.634461 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:05.634412    2582 projected.go:196] Error preparing data for projected volume kube-api-access-7vvj2 for pod openshift-network-diagnostics/network-check-target-jh728: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:41:05.634461 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:05.634458    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/204212cc-1c4c-4bfb-8b41-03a98042166c-kube-api-access-7vvj2 podName:204212cc-1c4c-4bfb-8b41-03a98042166c nodeName:}" failed. No retries permitted until 2026-10-30 00:41:21.63444509 +0000 UTC m=+34.326705347 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7vvj2" (UniqueName: "kubernetes.io/projected/204212cc-1c4c-4bfb-8b41-03a98042166c-kube-api-access-7vvj2") pod "network-check-target-jh728" (UID: "204212cc-1c4c-4bfb-8b41-03a98042166c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:41:05.835126 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:05.835023    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:41:05.835126 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:05.835049    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:41:05.835319 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:05.835174    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b4686" podUID="0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493"
Oct 30 00:41:05.835319 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:05.835268    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wk65n" podUID="bbef968a-7aa4-4e23-9278-3706e60d8c0b"
Oct 30 00:41:06.142531 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:06.142456    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tz5vn" event={"ID":"f5fade25-18f0-4e3f-978a-5bb39a356ab0","Type":"ContainerStarted","Data":"7a0b6a5c780d70924c4e0521cf1e240dd3f0a1ac457fc6bc700b1954d725caea"}
Oct 30 00:41:06.220702 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:06.220656    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tz5vn" podStartSLOduration=8.074400281 podStartE2EDuration="18.220642712s" totalImagesPullingTime="10.146242431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:40:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:06.206700974 +0000 UTC m=+18.898961272" watchObservedRunningTime="2026-10-30 00:41:06.220642712 +0000 UTC m=+18.912902992"
Oct 30 00:41:06.835322 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:06.835298    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:41:06.835452 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:06.835391    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh728" podUID="204212cc-1c4c-4bfb-8b41-03a98042166c"
Oct 30 00:41:07.836209 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:07.836178    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:41:07.836634 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:07.836315    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b4686" podUID="0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493"
Oct 30 00:41:07.836634 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:07.836369    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:41:07.836634 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:07.836465    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wk65n" podUID="bbef968a-7aa4-4e23-9278-3706e60d8c0b"
Oct 30 00:41:08.150131 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:08.150049    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7855k" event={"ID":"380df728-1bf4-4258-ba04-c31e1d28a3aa","Type":"ContainerStarted","Data":"ec3051b364dcc3b2d9b723dab16cd2f92cc244cf8bd7a99ac0a43bc49747a443"}
Oct 30 00:41:08.835405 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:08.835360    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:41:08.835596 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:08.835474    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh728" podUID="204212cc-1c4c-4bfb-8b41-03a98042166c"
Oct 30 00:41:09.834528 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:09.834492    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:41:09.835307 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:09.834603    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b4686" podUID="0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493"
Oct 30 00:41:09.835307 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:09.834658    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:41:09.835307 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:09.834746    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wk65n" podUID="bbef968a-7aa4-4e23-9278-3706e60d8c0b"
Oct 30 00:41:10.160056 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:10.159969    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7855k" event={"ID":"380df728-1bf4-4258-ba04-c31e1d28a3aa","Type":"ContainerStarted","Data":"4a64b2f1e84369f72e28eaab9083d7f94615007e98296f3f43b88403bc492f38"}
Oct 30 00:41:10.160970 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:10.160940    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:10.161122 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:10.161043    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:10.176026 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:10.176003    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:10.176306 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:10.176289    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:10.259716 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:10.259671    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7855k" podStartSLOduration=7.554302071 podStartE2EDuration="22.25965823s" totalImagesPullingTime="14.705356159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:10.259373903 +0000 UTC m=+22.951634182" watchObservedRunningTime="2026-10-30 00:41:10.25965823 +0000 UTC m=+22.951918508"
Oct 30 00:41:10.835541 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:10.835507    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:41:10.835899 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:10.835615    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh728" podUID="204212cc-1c4c-4bfb-8b41-03a98042166c"
Oct 30 00:41:11.162652 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:11.162582    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:41:11.348750 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:11.348716    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wk65n"]
Oct 30 00:41:11.348896 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:11.348845    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:41:11.348969 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:11.348951    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wk65n" podUID="bbef968a-7aa4-4e23-9278-3706e60d8c0b"
Oct 30 00:41:11.352171 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:11.352143    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jh728"]
Oct 30 00:41:11.352280 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:11.352235    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:41:11.352347 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:11.352328    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh728" podUID="204212cc-1c4c-4bfb-8b41-03a98042166c"
Oct 30 00:41:11.352746 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:11.352731    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b4686"]
Oct 30 00:41:11.352817 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:11.352808    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:41:11.352891 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:11.352877    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b4686" podUID="0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493"
Oct 30 00:41:12.164974 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:12.164947    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:41:12.834880 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:12.834847    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:41:12.835015 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:12.834940    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:41:12.835015 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:12.834968    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:41:12.835015 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:12.834950    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh728" podUID="204212cc-1c4c-4bfb-8b41-03a98042166c"
Oct 30 00:41:12.835141 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:12.835059    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wk65n" podUID="bbef968a-7aa4-4e23-9278-3706e60d8c0b"
Oct 30 00:41:12.835177 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:12.835140    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b4686" podUID="0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493"
Oct 30 00:41:13.133823 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:13.133740    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:14.835231 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:14.835200    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:41:14.835632 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:14.835312    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:41:14.835632 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:14.835311    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:41:14.835632 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:14.835312    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jh728" podUID="204212cc-1c4c-4bfb-8b41-03a98042166c"
Oct 30 00:41:14.835632 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:14.835378    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b4686" podUID="0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493"
Oct 30 00:41:14.835632 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:14.835499    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wk65n" podUID="bbef968a-7aa4-4e23-9278-3706e60d8c0b"
Oct 30 00:41:16.555929 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.555849    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeReady"
Oct 30 00:41:16.556330 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.555968    2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 00:41:16.603421 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.603395    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pq5xl"]
Oct 30 00:41:16.605738 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.605720    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pq5xl"
Oct 30 00:41:16.607810 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.607767    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f45784fb5-r5j44"]
Oct 30 00:41:16.611016 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.610988    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6q8k5"]
Oct 30 00:41:16.611200 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.611178    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:16.614424 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.614402    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dtn28"]
Oct 30 00:41:16.614535 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.614509    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6q8k5"
Oct 30 00:41:16.617110 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.617088    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s"]
Oct 30 00:41:16.617227 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.617153    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-dtn28"
Oct 30 00:41:16.619756 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.619738    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-654dd5d568-jdc79"]
Oct 30 00:41:16.619873 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.619857    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s"
Oct 30 00:41:16.622954 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.622933    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-j7px8"]
Oct 30 00:41:16.623059 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.623027    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:16.625615 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.625598    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jk65g"]
Oct 30 00:41:16.625721 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.625708    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-j7px8"
Oct 30 00:41:16.628467 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.628449    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"]
Oct 30 00:41:16.628577 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.628561    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:16.631512 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.631493    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs"]
Oct 30 00:41:16.631609 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.631599    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"
Oct 30 00:41:16.632335 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.632313    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 00:41:16.634130 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.634105    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"]
Oct 30 00:41:16.634281 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.634267    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs"
Oct 30 00:41:16.636816 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.636799    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-94c9fcc47-fdtjt"]
Oct 30 00:41:16.636922 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.636909    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:16.639895 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.639830    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xxvch"]
Oct 30 00:41:16.640011 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.639989    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:16.642614 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.642598    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nkb8p"]
Oct 30 00:41:16.642730 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.642714    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:16.645475 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.645456    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-9zmkk"]
Oct 30 00:41:16.645580 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.645568    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nkb8p"
Oct 30 00:41:16.646144 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.646127    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 00:41:16.646201 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.646127    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 00:41:16.649186 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.649169    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wq8t9"]
Oct 30 00:41:16.649288 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.649274    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9zmkk"
Oct 30 00:41:16.653869 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.652873    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hf7tk"]
Oct 30 00:41:16.653954 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.653918    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:16.657755 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.657555    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pq5xl"]
Oct 30 00:41:16.657755 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.657630    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hf7tk"
Oct 30 00:41:16.659673 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.659654    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"]
Oct 30 00:41:16.670813 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.670792    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6q8k5"]
Oct 30 00:41:16.671790 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.671769    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dtn28"]
Oct 30 00:41:16.672670 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.672652    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"]
Oct 30 00:41:16.681094 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.678038    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nkb8p"]
Oct 30 00:41:16.681094 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.678106    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jk65g"]
Oct 30 00:41:16.681094 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.678130    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-9zmkk"]
Oct 30 00:41:16.681094 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.679088    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f45784fb5-r5j44"]
Oct 30 00:41:16.681094 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.680729    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s"]
Oct 30 00:41:16.681372 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.681331    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-j7px8"]
Oct 30 00:41:16.703669 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.703643    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xxvch"]
Oct 30 00:41:16.708624 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.708602    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hf7tk"]
Oct 30 00:41:16.711620 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.711594    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75l4z\" (UniqueName: \"kubernetes.io/projected/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-kube-api-access-75l4z\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:16.711741 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.711634    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c48ee8c6-60b2-4229-a013-4199c44813e4-trusted-ca\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:16.711741 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.711661    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/433f04c0-a9ea-45c8-a2b7-245980607391-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs\" (UID: \"433f04c0-a9ea-45c8-a2b7-245980607391\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs"
Oct 30 00:41:16.711741 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.711719    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0142f4ce-76c5-4ca1-ac00-c2c334696905-serving-cert\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:16.711902 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.711754    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s72cg\" (UniqueName: \"kubernetes.io/projected/a2dadf26-b255-4281-91f4-98536d370d21-kube-api-access-s72cg\") pod \"kas-connection-checker-685895c7c9-dtn28\" (UID: \"a2dadf26-b255-4281-91f4-98536d370d21\") " pod="kube-system/kas-connection-checker-685895c7c9-dtn28"
Oct 30 00:41:16.711902 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.711803    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6a145873-a56a-4fe8-bdb5-dd0e6837b948-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j7px8\" (UID: \"6a145873-a56a-4fe8-bdb5-dd0e6837b948\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j7px8"
Oct 30 00:41:16.711902 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.711828    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmm4r\" (UniqueName: \"kubernetes.io/projected/59e5f12b-4a3e-449b-b285-17747780e1cf-kube-api-access-cmm4r\") pod \"service-ca-operator-544d749c8c-jbp7s\" (UID: \"59e5f12b-4a3e-449b-b285-17747780e1cf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s"
Oct 30 00:41:16.711902 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.711856    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0142f4ce-76c5-4ca1-ac00-c2c334696905-service-ca-bundle\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:16.711902 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.711885    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c48ee8c6-60b2-4229-a013-4199c44813e4-ca-trust-extracted\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:16.712171 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.711911    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0142f4ce-76c5-4ca1-ac00-c2c334696905-snapshots\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:16.712171 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.711941    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0142f4ce-76c5-4ca1-ac00-c2c334696905-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:16.712171 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.711966    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79mmb\" (UniqueName: \"kubernetes.io/projected/794aa402-53bf-43fc-b21b-a176bbbcd132-kube-api-access-79mmb\") pod \"cluster-samples-operator-6ddb6c8b94-gh5xc\" (UID: \"794aa402-53bf-43fc-b21b-a176bbbcd132\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"
Oct 30 00:41:16.712171 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.711993    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8xgb\" (UniqueName: \"kubernetes.io/projected/e05b9b53-5f5b-402e-8599-ce61481346e6-kube-api-access-l8xgb\") pod \"volume-data-source-validator-58c9f5f46d-nkb8p\" (UID: \"e05b9b53-5f5b-402e-8599-ce61481346e6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nkb8p"
Oct 30 00:41:16.712171 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712018    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/59e5f12b-4a3e-449b-b285-17747780e1cf-config\") pod \"service-ca-operator-544d749c8c-jbp7s\" (UID: \"59e5f12b-4a3e-449b-b285-17747780e1cf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s"
Oct 30 00:41:16.712171 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712041    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-config-volume\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:16.712171 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712107    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c48ee8c6-60b2-4229-a013-4199c44813e4-image-registry-private-configuration\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:16.712171 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712135    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7glf\" (UniqueName: \"kubernetes.io/projected/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-kube-api-access-v7glf\") pod \"cluster-monitoring-operator-884576c57-lfnrl\" (UID: \"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:16.712171 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712163    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:16.712603 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712188    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-lfnrl\" (UID: \"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:16.712603 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712215    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-stats-auth\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:16.712603 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712250    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/433f04c0-a9ea-45c8-a2b7-245980607391-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs\" (UID: \"433f04c0-a9ea-45c8-a2b7-245980607391\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs"
Oct 30 00:41:16.712603 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712276    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0142f4ce-76c5-4ca1-ac00-c2c334696905-tmp\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:16.712603 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712300    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-bound-sa-token\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:16.712603 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712325    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8e4dbba6-28af-46b3-ac90-37ce437fd318-trusted-ca\") pod \"console-operator-7cd56b8dbc-jk65g\" (UID: \"8e4dbba6-28af-46b3-ac90-37ce437fd318\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:16.712603 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712351    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g77n6\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-kube-api-access-g77n6\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:16.712603 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712377    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e4dbba6-28af-46b3-ac90-37ce437fd318-config\") pod \"console-operator-7cd56b8dbc-jk65g\" (UID: \"8e4dbba6-28af-46b3-ac90-37ce437fd318\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:16.712603 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712401    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gh5xc\" (UID: \"794aa402-53bf-43fc-b21b-a176bbbcd132\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"
Oct 30 00:41:16.712603 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712429    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:16.712603 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712469    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/564f9ba3-d19e-4cac-9021-81939eb8e631-trusted-ca\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:16.712603 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712495    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:16.712603 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712522    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-default-certificate\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:16.712603 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712548    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:16.712603 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712576    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45zv7\" (UniqueName: \"kubernetes.io/projected/0142f4ce-76c5-4ca1-ac00-c2c334696905-kube-api-access-45zv7\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:16.713364 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712600    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e4dbba6-28af-46b3-ac90-37ce437fd318-serving-cert\") pod \"console-operator-7cd56b8dbc-jk65g\" (UID: \"8e4dbba6-28af-46b3-ac90-37ce437fd318\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:16.713364 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712625    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/564f9ba3-d19e-4cac-9021-81939eb8e631-image-registry-private-configuration\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:16.713364 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712659    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sljzz\" (UniqueName: \"kubernetes.io/projected/8e4dbba6-28af-46b3-ac90-37ce437fd318-kube-api-access-sljzz\") pod \"console-operator-7cd56b8dbc-jk65g\" (UID: \"8e4dbba6-28af-46b3-ac90-37ce437fd318\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:16.713364 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712684    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpx52\" (UniqueName: \"kubernetes.io/projected/fc3514a7-8b72-40ff-b45a-1c2005b59360-kube-api-access-cpx52\") pod \"kas-connection-checker-685895c7c9-pq5xl\" (UID: \"fc3514a7-8b72-40ff-b45a-1c2005b59360\") " pod="kube-system/kas-connection-checker-685895c7c9-pq5xl"
Oct 30 00:41:16.713364 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712729    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-tmp-dir\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:16.713364 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712756    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/564f9ba3-d19e-4cac-9021-81939eb8e631-installation-pull-secrets\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:16.713364 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712782    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/59e5f12b-4a3e-449b-b285-17747780e1cf-serving-cert\") pod \"service-ca-operator-544d749c8c-jbp7s\" (UID: \"59e5f12b-4a3e-449b-b285-17747780e1cf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s"
Oct 30 00:41:16.713364 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712806    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r64gt\" (UniqueName: \"kubernetes.io/projected/76b2bcb8-636c-46b5-8dfb-95831470349e-kube-api-access-r64gt\") pod \"network-check-source-6f78d59f99-9zmkk\" (UID: \"76b2bcb8-636c-46b5-8dfb-95831470349e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9zmkk"
Oct 30 00:41:16.713364 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712833    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c48ee8c6-60b2-4229-a013-4199c44813e4-installation-pull-secrets\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:16.713364 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712864    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78557\" (UniqueName: \"kubernetes.io/projected/d95535e9-e677-463a-a662-7e8045c71d9b-kube-api-access-78557\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:16.713364 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712893    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:16.713364 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712917    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lfnrl\" (UID: \"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:16.713364 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712943    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8s6t\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-kube-api-access-d8s6t\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:16.713364 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712969    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q94d\" (UniqueName: \"kubernetes.io/projected/433f04c0-a9ea-45c8-a2b7-245980607391-kube-api-access-4q94d\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs\" (UID: \"433f04c0-a9ea-45c8-a2b7-245980607391\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs"
Oct 30 00:41:16.713364 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.712998    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-certificates\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:16.714115 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.713023    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j7px8\" (UID: \"6a145873-a56a-4fe8-bdb5-dd0e6837b948\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j7px8"
Oct 30 00:41:16.714115 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.713053    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gh5xc\" (UID: \"794aa402-53bf-43fc-b21b-a176bbbcd132\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"
Oct 30 00:41:16.714115 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.713107    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-certificates\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:16.714115 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.713302    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs"]
Oct 30 00:41:16.714115 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.713334    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/564f9ba3-d19e-4cac-9021-81939eb8e631-ca-trust-extracted\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:16.714115 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.713378    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tkrk\" (UniqueName: \"kubernetes.io/projected/4509811e-30a0-4c8b-b21e-58826bd1f6c6-kube-api-access-5tkrk\") pod \"kas-connection-checker-685895c7c9-6q8k5\" (UID: \"4509811e-30a0-4c8b-b21e-58826bd1f6c6\") " pod="kube-system/kas-connection-checker-685895c7c9-6q8k5"
Oct 30 00:41:16.714115 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.713421    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-bound-sa-token\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:16.714530 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.714511    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wq8t9"]
Oct 30 00:41:16.716534 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.716461    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-94c9fcc47-fdtjt"]
Oct 30 00:41:16.720356 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.720336    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-654dd5d568-jdc79"]
Oct 30 00:41:16.734736 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.734710    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 00:41:16.734736 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.734727    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 00:41:16.734918 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.734759    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 00:41:16.735004 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.734983    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 00:41:16.735119 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.735019    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-lxq9t\""
Oct 30 00:41:16.735119 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.735047    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 00:41:16.735119 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.735054    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 00:41:16.735119 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.735057    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 00:41:16.735119 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.735055    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6kdk2\""
Oct 30 00:41:16.735119 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.735051    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 00:41:16.735119 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.735110    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l8pvs\""
Oct 30 00:41:16.735119 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.735117    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 00:41:16.735401 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.735114    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 00:41:16.735401 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.735133    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 00:41:16.735401 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.735131    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 00:41:16.735401 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.735136    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 00:41:16.735401 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.735167    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 00:41:16.735401 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.735214    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-x7m5x\""
Oct 30 00:41:16.739680 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.739657    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:41:16.739947 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.739915    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 00:41:16.740010 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.740001    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 00:41:16.742411 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.740616    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 00:41:16.742411 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.740685    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 00:41:16.742411 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.740738    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 00:41:16.742411 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.740808    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 00:41:16.742411 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.740852    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 00:41:16.742411 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.740909    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 00:41:16.742411 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.741059    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 00:41:16.742411 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.741170    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:41:16.742411 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.741196    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 00:41:16.742411 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.741296    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 00:41:16.743676 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.743577    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 00:41:16.744403 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.744386    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:41:16.748710 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.748696    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 00:41:16.748777 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.748731    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 00:41:16.748777 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.748754    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 00:41:16.748777 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.748765    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 00:41:16.748886 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.748765    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 00:41:16.749037 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.749021    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:41:16.749100 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.749039    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 00:41:16.749100 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.749054    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 00:41:16.749166 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.749057    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 00:41:16.749166 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.749087    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 00:41:16.749273 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.749206    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:41:16.753522 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.753510    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 00:41:16.754043 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.754030    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 00:41:16.754093 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.754032    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 00:41:16.762590 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.762568    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 00:41:16.768137 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.768123    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 00:41:16.768340 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.768325    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 00:41:16.814762 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.814703    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e4dbba6-28af-46b3-ac90-37ce437fd318-config\") pod \"console-operator-7cd56b8dbc-jk65g\" (UID: \"8e4dbba6-28af-46b3-ac90-37ce437fd318\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:16.814762 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.814733    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gh5xc\" (UID: \"794aa402-53bf-43fc-b21b-a176bbbcd132\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"
Oct 30 00:41:16.814762 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.814749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:16.814973 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.814785    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/564f9ba3-d19e-4cac-9021-81939eb8e631-trusted-ca\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:16.814973 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.814811    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:16.814973 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:16.814858    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 00:41:16.814973 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.814863    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-default-certificate\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:16.814973 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:16.814894    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:41:16.814973 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:16.814902    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle podName:d95535e9-e677-463a-a662-7e8045c71d9b nodeName:}" failed. No retries permitted until 2026-10-30 00:41:17.314886465 +0000 UTC m=+30.007146721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle") pod "router-default-94c9fcc47-fdtjt" (UID: "d95535e9-e677-463a-a662-7e8045c71d9b") : configmap references non-existent config key: service-ca.crt
Oct 30 00:41:16.814973 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:16.814921    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-tls podName:794aa402-53bf-43fc-b21b-a176bbbcd132 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:17.314914965 +0000 UTC m=+30.007175221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gh5xc" (UID: "794aa402-53bf-43fc-b21b-a176bbbcd132") : secret "samples-operator-tls" not found
Oct 30 00:41:16.814973 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.814935    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:16.814973 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.814963    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45zv7\" (UniqueName: \"kubernetes.io/projected/0142f4ce-76c5-4ca1-ac00-c2c334696905-kube-api-access-45zv7\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:16.815379 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:16.814907    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f45784fb5-r5j44: secret "image-registry-tls" not found
Oct 30 00:41:16.815379 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.814989    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e4dbba6-28af-46b3-ac90-37ce437fd318-serving-cert\") pod \"console-operator-7cd56b8dbc-jk65g\" (UID: \"8e4dbba6-28af-46b3-ac90-37ce437fd318\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:16.815379 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815017    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/564f9ba3-d19e-4cac-9021-81939eb8e631-image-registry-private-configuration\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:16.815379 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:16.815031    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls podName:564f9ba3-d19e-4cac-9021-81939eb8e631 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:17.315014363 +0000 UTC m=+30.007274653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls") pod "image-registry-5f45784fb5-r5j44" (UID: "564f9ba3-d19e-4cac-9021-81939eb8e631") : secret "image-registry-tls" not found
Oct 30 00:41:16.815379 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815067    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sljzz\" (UniqueName: \"kubernetes.io/projected/8e4dbba6-28af-46b3-ac90-37ce437fd318-kube-api-access-sljzz\") pod \"console-operator-7cd56b8dbc-jk65g\" (UID: \"8e4dbba6-28af-46b3-ac90-37ce437fd318\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:16.815379 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815119    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpx52\" (UniqueName: \"kubernetes.io/projected/fc3514a7-8b72-40ff-b45a-1c2005b59360-kube-api-access-cpx52\") pod \"kas-connection-checker-685895c7c9-pq5xl\" (UID: \"fc3514a7-8b72-40ff-b45a-1c2005b59360\") " pod="kube-system/kas-connection-checker-685895c7c9-pq5xl"
Oct 30 00:41:16.815379 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-tmp-dir\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:16.815379 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815188    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/564f9ba3-d19e-4cac-9021-81939eb8e631-installation-pull-secrets\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:16.815379 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815214    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/59e5f12b-4a3e-449b-b285-17747780e1cf-serving-cert\") pod \"service-ca-operator-544d749c8c-jbp7s\" (UID: \"59e5f12b-4a3e-449b-b285-17747780e1cf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s"
Oct 30 00:41:16.815379 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r64gt\" (UniqueName: \"kubernetes.io/projected/76b2bcb8-636c-46b5-8dfb-95831470349e-kube-api-access-r64gt\") pod \"network-check-source-6f78d59f99-9zmkk\" (UID: \"76b2bcb8-636c-46b5-8dfb-95831470349e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9zmkk"
Oct 30 00:41:16.815379 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815267    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c48ee8c6-60b2-4229-a013-4199c44813e4-installation-pull-secrets\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:16.815379 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815296    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78557\" (UniqueName: \"kubernetes.io/projected/d95535e9-e677-463a-a662-7e8045c71d9b-kube-api-access-78557\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:16.815379 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815324    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:16.815379 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815349    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lfnrl\" (UID: \"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:16.815379 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815378    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54t8p\" (UniqueName: \"kubernetes.io/projected/a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55-kube-api-access-54t8p\") pod \"ingress-canary-hf7tk\" (UID: \"a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55\") " pod="openshift-ingress-canary/ingress-canary-hf7tk"
Oct 30 00:41:16.815992 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815407    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8s6t\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-kube-api-access-d8s6t\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:16.815992 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815434    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q94d\" (UniqueName: \"kubernetes.io/projected/433f04c0-a9ea-45c8-a2b7-245980607391-kube-api-access-4q94d\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs\" (UID: \"433f04c0-a9ea-45c8-a2b7-245980607391\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs"
Oct 30 00:41:16.815992 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815464    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-certificates\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:16.815992 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815489    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j7px8\" (UID: \"6a145873-a56a-4fe8-bdb5-dd0e6837b948\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j7px8"
Oct 30 00:41:16.815992 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815515    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gh5xc\" (UID: \"794aa402-53bf-43fc-b21b-a176bbbcd132\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"
Oct 30 00:41:16.815992 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815546    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-certificates\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:16.815992 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815588    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/564f9ba3-d19e-4cac-9021-81939eb8e631-ca-trust-extracted\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:16.815992 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815615    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tkrk\" (UniqueName: \"kubernetes.io/projected/4509811e-30a0-4c8b-b21e-58826bd1f6c6-kube-api-access-5tkrk\") pod \"kas-connection-checker-685895c7c9-6q8k5\" (UID: \"4509811e-30a0-4c8b-b21e-58826bd1f6c6\") " pod="kube-system/kas-connection-checker-685895c7c9-6q8k5"
Oct 30 00:41:16.815992 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815626    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e4dbba6-28af-46b3-ac90-37ce437fd318-config\") pod \"console-operator-7cd56b8dbc-jk65g\" (UID: \"8e4dbba6-28af-46b3-ac90-37ce437fd318\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:16.815992 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815639    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-bound-sa-token\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:16.815992 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815664    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55-cert\") pod \"ingress-canary-hf7tk\" (UID: \"a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55\") " pod="openshift-ingress-canary/ingress-canary-hf7tk"
Oct 30 00:41:16.815992 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815702    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75l4z\" (UniqueName: \"kubernetes.io/projected/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-kube-api-access-75l4z\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:16.815992 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815712    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/564f9ba3-d19e-4cac-9021-81939eb8e631-trusted-ca\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:16.815992 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815728    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c48ee8c6-60b2-4229-a013-4199c44813e4-trusted-ca\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:16.815992 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815753    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/433f04c0-a9ea-45c8-a2b7-245980607391-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs\" (UID: \"433f04c0-a9ea-45c8-a2b7-245980607391\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs"
Oct 30 00:41:16.816572 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815781    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0142f4ce-76c5-4ca1-ac00-c2c334696905-serving-cert\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:16.816572 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815805    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s72cg\" (UniqueName: \"kubernetes.io/projected/a2dadf26-b255-4281-91f4-98536d370d21-kube-api-access-s72cg\") pod \"kas-connection-checker-685895c7c9-dtn28\" (UID: \"a2dadf26-b255-4281-91f4-98536d370d21\") " pod="kube-system/kas-connection-checker-685895c7c9-dtn28"
Oct 30 00:41:16.816572 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815841    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6a145873-a56a-4fe8-bdb5-dd0e6837b948-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j7px8\" (UID: \"6a145873-a56a-4fe8-bdb5-dd0e6837b948\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j7px8"
Oct 30 00:41:16.816572 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815862    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmm4r\" (UniqueName: \"kubernetes.io/projected/59e5f12b-4a3e-449b-b285-17747780e1cf-kube-api-access-cmm4r\") pod \"service-ca-operator-544d749c8c-jbp7s\" (UID: \"59e5f12b-4a3e-449b-b285-17747780e1cf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s"
Oct 30 00:41:16.816572 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815883    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0142f4ce-76c5-4ca1-ac00-c2c334696905-service-ca-bundle\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:16.816572 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815891    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-tmp-dir\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:16.816572 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815912    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c48ee8c6-60b2-4229-a013-4199c44813e4-ca-trust-extracted\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:16.816572 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815935    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0142f4ce-76c5-4ca1-ac00-c2c334696905-snapshots\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:16.816572 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815963    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0142f4ce-76c5-4ca1-ac00-c2c334696905-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:16.816572 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.815985    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79mmb\" (UniqueName: \"kubernetes.io/projected/794aa402-53bf-43fc-b21b-a176bbbcd132-kube-api-access-79mmb\") pod \"cluster-samples-operator-6ddb6c8b94-gh5xc\" (UID: \"794aa402-53bf-43fc-b21b-a176bbbcd132\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"
Oct 30 00:41:16.816572 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.816007    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8xgb\" (UniqueName: \"kubernetes.io/projected/e05b9b53-5f5b-402e-8599-ce61481346e6-kube-api-access-l8xgb\") pod \"volume-data-source-validator-58c9f5f46d-nkb8p\" (UID: \"e05b9b53-5f5b-402e-8599-ce61481346e6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nkb8p"
Oct 30 00:41:16.816572 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.816031    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/59e5f12b-4a3e-449b-b285-17747780e1cf-config\") pod \"service-ca-operator-544d749c8c-jbp7s\" (UID: \"59e5f12b-4a3e-449b-b285-17747780e1cf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s"
Oct 30 00:41:16.816572 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.816051    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-config-volume\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:16.816572 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.816092    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c48ee8c6-60b2-4229-a013-4199c44813e4-image-registry-private-configuration\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:16.816572 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.816116    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7glf\" (UniqueName: \"kubernetes.io/projected/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-kube-api-access-v7glf\") pod \"cluster-monitoring-operator-884576c57-lfnrl\" (UID: \"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:16.816572 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.816145    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:16.817276 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.816180    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-lfnrl\" (UID: \"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:16.817276 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:16.816437    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:41:16.817276 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:16.816596    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls podName:cee8ca5c-ba97-47b5-b22f-2e161bb93fdc nodeName:}" failed. No retries permitted until 2026-10-30 00:41:17.31658416 +0000 UTC m=+30.008844419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls") pod "dns-default-wq8t9" (UID: "cee8ca5c-ba97-47b5-b22f-2e161bb93fdc") : secret "dns-default-metrics-tls" not found
Oct 30 00:41:16.817276 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:16.816716    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 00:41:16.817276 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:16.816773    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs podName:d95535e9-e677-463a-a662-7e8045c71d9b nodeName:}" failed. No retries permitted until 2026-10-30 00:41:17.316758092 +0000 UTC m=+30.009018352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs") pod "router-default-94c9fcc47-fdtjt" (UID: "d95535e9-e677-463a-a662-7e8045c71d9b") : secret "router-metrics-certs-default" not found
Oct 30 00:41:16.817276 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.817109    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0142f4ce-76c5-4ca1-ac00-c2c334696905-snapshots\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:16.817276 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:16.817179    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 00:41:16.817276 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.817208    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/59e5f12b-4a3e-449b-b285-17747780e1cf-config\") pod \"service-ca-operator-544d749c8c-jbp7s\" (UID: \"59e5f12b-4a3e-449b-b285-17747780e1cf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s"
Oct 30 00:41:16.817276 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:16.817221    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert podName:6a145873-a56a-4fe8-bdb5-dd0e6837b948 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:17.317206933 +0000 UTC m=+30.009467192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j7px8" (UID: "6a145873-a56a-4fe8-bdb5-dd0e6837b948") : secret "networking-console-plugin-cert" not found
Oct 30 00:41:16.817697 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.817305    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0142f4ce-76c5-4ca1-ac00-c2c334696905-service-ca-bundle\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:16.817697 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.817416    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6a145873-a56a-4fe8-bdb5-dd0e6837b948-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j7px8\" (UID: \"6a145873-a56a-4fe8-bdb5-dd0e6837b948\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j7px8"
Oct 30 00:41:16.817697 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.817457    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-stats-auth\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:16.817697 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.817500    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/433f04c0-a9ea-45c8-a2b7-245980607391-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs\" (UID: \"433f04c0-a9ea-45c8-a2b7-245980607391\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs"
Oct 30 00:41:16.817697 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.817644    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c48ee8c6-60b2-4229-a013-4199c44813e4-trusted-ca\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:16.817930 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:16.816545    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 00:41:16.817930 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.817862    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gh5xc\" (UID: \"794aa402-53bf-43fc-b21b-a176bbbcd132\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"
Oct 30 00:41:16.817930 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:16.817899    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls podName:2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc nodeName:}" failed. No retries permitted until 2026-10-30 00:41:17.317881216 +0000 UTC m=+30.010141477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lfnrl" (UID: "2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc") : secret "cluster-monitoring-operator-tls" not found
Oct 30 00:41:16.818062 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:16.818041    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:41:16.818062 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:16.818054    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-654dd5d568-jdc79: secret "image-registry-tls" not found
Oct 30 00:41:16.818224 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:16.818116    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls podName:c48ee8c6-60b2-4229-a013-4199c44813e4 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:17.318100469 +0000 UTC m=+30.010360769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls") pod "image-registry-654dd5d568-jdc79" (UID: "c48ee8c6-60b2-4229-a013-4199c44813e4") : secret "image-registry-tls" not found
Oct 30 00:41:16.818286 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.818262    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c48ee8c6-60b2-4229-a013-4199c44813e4-ca-trust-extracted\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:16.819028 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.818575    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-certificates\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:16.819028 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.818603    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-config-volume\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:16.819028 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.818039    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-certificates\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:16.819028 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.818982    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0142f4ce-76c5-4ca1-ac00-c2c334696905-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:16.819311 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.819231    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-lfnrl\" (UID: \"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:16.819656 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.819633    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0142f4ce-76c5-4ca1-ac00-c2c334696905-tmp\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:16.819747 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.819680    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-bound-sa-token\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:16.819747 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.819716    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/564f9ba3-d19e-4cac-9021-81939eb8e631-ca-trust-extracted\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:16.819747 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.819741    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8e4dbba6-28af-46b3-ac90-37ce437fd318-trusted-ca\") pod \"console-operator-7cd56b8dbc-jk65g\" (UID: \"8e4dbba6-28af-46b3-ac90-37ce437fd318\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:16.819903 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.819771    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g77n6\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-kube-api-access-g77n6\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:16.819903 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.819804    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/433f04c0-a9ea-45c8-a2b7-245980607391-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs\" (UID: \"433f04c0-a9ea-45c8-a2b7-245980607391\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs"
Oct 30 00:41:16.820340 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.820315    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/564f9ba3-d19e-4cac-9021-81939eb8e631-installation-pull-secrets\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:16.820425 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.820404    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0142f4ce-76c5-4ca1-ac00-c2c334696905-tmp\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:16.820715 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.820688    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/564f9ba3-d19e-4cac-9021-81939eb8e631-image-registry-private-configuration\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:16.821112 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.821090    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8e4dbba6-28af-46b3-ac90-37ce437fd318-trusted-ca\") pod \"console-operator-7cd56b8dbc-jk65g\" (UID: \"8e4dbba6-28af-46b3-ac90-37ce437fd318\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:16.821194 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.821149    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-default-certificate\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:16.821431 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.821403    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e4dbba6-28af-46b3-ac90-37ce437fd318-serving-cert\") pod \"console-operator-7cd56b8dbc-jk65g\" (UID: \"8e4dbba6-28af-46b3-ac90-37ce437fd318\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:16.821511 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.821493    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0142f4ce-76c5-4ca1-ac00-c2c334696905-serving-cert\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:16.821665 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.821639    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/59e5f12b-4a3e-449b-b285-17747780e1cf-serving-cert\") pod \"service-ca-operator-544d749c8c-jbp7s\" (UID: \"59e5f12b-4a3e-449b-b285-17747780e1cf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s"
Oct 30 00:41:16.822227 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.822206    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/433f04c0-a9ea-45c8-a2b7-245980607391-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs\" (UID: \"433f04c0-a9ea-45c8-a2b7-245980607391\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs"
Oct 30 00:41:16.822610 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.822587    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-stats-auth\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:16.823021 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.823002    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c48ee8c6-60b2-4229-a013-4199c44813e4-installation-pull-secrets\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:16.823543 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.823522    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c48ee8c6-60b2-4229-a013-4199c44813e4-image-registry-private-configuration\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:16.835478 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.835459    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:41:16.835549 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.835522    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:41:16.835549 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.835543    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:41:16.848015 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.847992    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45zv7\" (UniqueName: \"kubernetes.io/projected/0142f4ce-76c5-4ca1-ac00-c2c334696905-kube-api-access-45zv7\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:16.849377 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.849335    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpx52\" (UniqueName: \"kubernetes.io/projected/fc3514a7-8b72-40ff-b45a-1c2005b59360-kube-api-access-cpx52\") pod \"kas-connection-checker-685895c7c9-pq5xl\" (UID: \"fc3514a7-8b72-40ff-b45a-1c2005b59360\") " pod="kube-system/kas-connection-checker-685895c7c9-pq5xl"
Oct 30 00:41:16.849815 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.849786    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s72cg\" (UniqueName: \"kubernetes.io/projected/a2dadf26-b255-4281-91f4-98536d370d21-kube-api-access-s72cg\") pod \"kas-connection-checker-685895c7c9-dtn28\" (UID: \"a2dadf26-b255-4281-91f4-98536d370d21\") " pod="kube-system/kas-connection-checker-685895c7c9-dtn28"
Oct 30 00:41:16.850314 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.850100    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79mmb\" (UniqueName: \"kubernetes.io/projected/794aa402-53bf-43fc-b21b-a176bbbcd132-kube-api-access-79mmb\") pod \"cluster-samples-operator-6ddb6c8b94-gh5xc\" (UID: \"794aa402-53bf-43fc-b21b-a176bbbcd132\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"
Oct 30 00:41:16.850867 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.850846    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-bound-sa-token\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:16.851236 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.851213    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmm4r\" (UniqueName: \"kubernetes.io/projected/59e5f12b-4a3e-449b-b285-17747780e1cf-kube-api-access-cmm4r\") pod \"service-ca-operator-544d749c8c-jbp7s\" (UID: \"59e5f12b-4a3e-449b-b285-17747780e1cf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s"
Oct 30 00:41:16.852093 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.852026    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8s6t\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-kube-api-access-d8s6t\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:16.852533 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.852511    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g77n6\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-kube-api-access-g77n6\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:16.852614 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.852571    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7glf\" (UniqueName: \"kubernetes.io/projected/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-kube-api-access-v7glf\") pod \"cluster-monitoring-operator-884576c57-lfnrl\" (UID: \"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:16.852779 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.852748    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78557\" (UniqueName: \"kubernetes.io/projected/d95535e9-e677-463a-a662-7e8045c71d9b-kube-api-access-78557\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:16.853017 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.852954    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 00:41:16.853458 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.853435    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-z4qxk\""
Oct 30 00:41:16.853865 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.853814    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tkrk\" (UniqueName: \"kubernetes.io/projected/4509811e-30a0-4c8b-b21e-58826bd1f6c6-kube-api-access-5tkrk\") pod \"kas-connection-checker-685895c7c9-6q8k5\" (UID: \"4509811e-30a0-4c8b-b21e-58826bd1f6c6\") " pod="kube-system/kas-connection-checker-685895c7c9-6q8k5"
Oct 30 00:41:16.853955 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.853874    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sljzz\" (UniqueName: \"kubernetes.io/projected/8e4dbba6-28af-46b3-ac90-37ce437fd318-kube-api-access-sljzz\") pod \"console-operator-7cd56b8dbc-jk65g\" (UID: \"8e4dbba6-28af-46b3-ac90-37ce437fd318\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:16.854610 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.854252    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wwjm6\""
Oct 30 00:41:16.854610 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.854339    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 00:41:16.854748 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.854608    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wm8np\""
Oct 30 00:41:16.855191 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.855171    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r64gt\" (UniqueName: \"kubernetes.io/projected/76b2bcb8-636c-46b5-8dfb-95831470349e-kube-api-access-r64gt\") pod \"network-check-source-6f78d59f99-9zmkk\" (UID: \"76b2bcb8-636c-46b5-8dfb-95831470349e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9zmkk"
Oct 30 00:41:16.855352 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.855332    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8xgb\" (UniqueName: \"kubernetes.io/projected/e05b9b53-5f5b-402e-8599-ce61481346e6-kube-api-access-l8xgb\") pod \"volume-data-source-validator-58c9f5f46d-nkb8p\" (UID: \"e05b9b53-5f5b-402e-8599-ce61481346e6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nkb8p"
Oct 30 00:41:16.856315 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.856294    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75l4z\" (UniqueName: \"kubernetes.io/projected/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-kube-api-access-75l4z\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:16.856610 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.856595    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-bound-sa-token\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:16.856980 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.856963    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q94d\" (UniqueName: \"kubernetes.io/projected/433f04c0-a9ea-45c8-a2b7-245980607391-kube-api-access-4q94d\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs\" (UID: \"433f04c0-a9ea-45c8-a2b7-245980607391\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs"
Oct 30 00:41:16.903466 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.903433    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-77dcb"
Oct 30 00:41:16.903627 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.903563    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:41:16.903957 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.903941    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-77dcb"
Oct 30 00:41:16.920746 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.920724    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54t8p\" (UniqueName: \"kubernetes.io/projected/a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55-kube-api-access-54t8p\") pod \"ingress-canary-hf7tk\" (UID: \"a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55\") " pod="openshift-ingress-canary/ingress-canary-hf7tk"
Oct 30 00:41:16.920814 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.920762    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55-cert\") pod \"ingress-canary-hf7tk\" (UID: \"a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55\") " pod="openshift-ingress-canary/ingress-canary-hf7tk"
Oct 30 00:41:16.920965 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.920951    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pq5xl"
Oct 30 00:41:16.922931 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.922915    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55-cert\") pod \"ingress-canary-hf7tk\" (UID: \"a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55\") " pod="openshift-ingress-canary/ingress-canary-hf7tk"
Oct 30 00:41:16.935517 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.935499    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6q8k5"
Oct 30 00:41:16.941552 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.941530    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-dtn28"
Oct 30 00:41:16.945958 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.945938    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54t8p\" (UniqueName: \"kubernetes.io/projected/a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55-kube-api-access-54t8p\") pod \"ingress-canary-hf7tk\" (UID: \"a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55\") " pod="openshift-ingress-canary/ingress-canary-hf7tk"
Oct 30 00:41:16.948733 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.948711    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s"
Oct 30 00:41:16.971700 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:16.971665    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:17.012831 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:17.011962    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs"
Oct 30 00:41:17.017255 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:17.016745    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:17.026895 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:17.026872    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nkb8p"
Oct 30 00:41:17.034441 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:17.033230    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9zmkk"
Oct 30 00:41:17.054598 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:17.044006    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hf7tk"
Oct 30 00:41:17.236956 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:17.236929    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pq5xl"]
Oct 30 00:41:17.241213 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:41:17.241187    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3514a7_8b72_40ff_b45a_1c2005b59360.slice/crio-a4be861396077b7336d341d30d02aa1c3b1d68b59eabd0533b5a32bf58a784fc WatchSource:0}: Error finding container a4be861396077b7336d341d30d02aa1c3b1d68b59eabd0533b5a32bf58a784fc: Status 404 returned error can't find the container with id a4be861396077b7336d341d30d02aa1c3b1d68b59eabd0533b5a32bf58a784fc
Oct 30 00:41:17.253825 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:17.253803    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6q8k5"]
Oct 30 00:41:17.259087 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:41:17.259050    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4509811e_30a0_4c8b_b21e_58826bd1f6c6.slice/crio-53018d64b73fdb4a068c8ca329984997d649894a8bd29be62d2a0095309671b4 WatchSource:0}: Error finding container 53018d64b73fdb4a068c8ca329984997d649894a8bd29be62d2a0095309671b4: Status 404 returned error can't find the container with id 53018d64b73fdb4a068c8ca329984997d649894a8bd29be62d2a0095309671b4
Oct 30 00:41:17.275353 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:17.275286    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dtn28"]
Oct 30 00:41:17.278522 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:41:17.278496    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2dadf26_b255_4281_91f4_98536d370d21.slice/crio-3cc619806395200599c333ff4d975468fa2183cea96e8de9ac4cdbc083843bb3 WatchSource:0}: Error finding container 3cc619806395200599c333ff4d975468fa2183cea96e8de9ac4cdbc083843bb3: Status 404 returned error can't find the container with id 3cc619806395200599c333ff4d975468fa2183cea96e8de9ac4cdbc083843bb3
Oct 30 00:41:17.289108 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:17.289087    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s"]
Oct 30 00:41:17.295514 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:41:17.295484    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e5f12b_4a3e_449b_b285_17747780e1cf.slice/crio-60513a52513278b2ffcb4c04eeda28a50d46522ccefecac86d43bf4faa41f6ee WatchSource:0}: Error finding container 60513a52513278b2ffcb4c04eeda28a50d46522ccefecac86d43bf4faa41f6ee: Status 404 returned error can't find the container with id 60513a52513278b2ffcb4c04eeda28a50d46522ccefecac86d43bf4faa41f6ee
Oct 30 00:41:17.333177 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:17.333147    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:17.333283 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:17.333194    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:17.333283 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:17.333262    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:17.333398 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:17.333289    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lfnrl\" (UID: \"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:17.333398 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:17.333306    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:41:17.333398 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:17.333329    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f45784fb5-r5j44: secret "image-registry-tls" not found
Oct 30 00:41:17.333398 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:17.333388    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls podName:564f9ba3-d19e-4cac-9021-81939eb8e631 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:18.333369408 +0000 UTC m=+31.025629683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls") pod "image-registry-5f45784fb5-r5j44" (UID: "564f9ba3-d19e-4cac-9021-81939eb8e631") : secret "image-registry-tls" not found
Oct 30 00:41:17.333598 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:17.333407    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 00:41:17.333598 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:17.333456    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert podName:6a145873-a56a-4fe8-bdb5-dd0e6837b948 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:18.333441264 +0000 UTC m=+31.025701538 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j7px8" (UID: "6a145873-a56a-4fe8-bdb5-dd0e6837b948") : secret "networking-console-plugin-cert" not found
Oct 30 00:41:17.333598 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:17.333510    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 00:41:17.333598 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:17.333542    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs podName:d95535e9-e677-463a-a662-7e8045c71d9b nodeName:}" failed. No retries permitted until 2026-10-30 00:41:18.333531748 +0000 UTC m=+31.025792020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs") pod "router-default-94c9fcc47-fdtjt" (UID: "d95535e9-e677-463a-a662-7e8045c71d9b") : secret "router-metrics-certs-default" not found
Oct 30 00:41:17.333786 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:17.333603    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:41:17.333786 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:17.333614    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-654dd5d568-jdc79: secret "image-registry-tls" not found
Oct 30 00:41:17.333786 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:17.333643    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls podName:c48ee8c6-60b2-4229-a013-4199c44813e4 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:18.333633418 +0000 UTC m=+31.025893716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls") pod "image-registry-654dd5d568-jdc79" (UID: "c48ee8c6-60b2-4229-a013-4199c44813e4") : secret "image-registry-tls" not found
Oct 30 00:41:17.333786 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:17.333691    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 00:41:17.333786 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:17.333325    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j7px8\" (UID: \"6a145873-a56a-4fe8-bdb5-dd0e6837b948\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j7px8"
Oct 30 00:41:17.333786 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:17.333720    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls podName:2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc nodeName:}" failed. No retries permitted until 2026-10-30 00:41:18.333711424 +0000 UTC m=+31.025971682 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lfnrl" (UID: "2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc") : secret "cluster-monitoring-operator-tls" not found
Oct 30 00:41:17.334061 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:17.333828    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:17.334061 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:17.333892    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gh5xc\" (UID: \"794aa402-53bf-43fc-b21b-a176bbbcd132\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"
Oct 30 00:41:17.334061 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:17.333923    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:17.334061 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:17.334044    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 00:41:17.334266 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:17.334160    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:41:17.334266 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:17.334200    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls podName:cee8ca5c-ba97-47b5-b22f-2e161bb93fdc nodeName:}" failed. No retries permitted until 2026-10-30 00:41:18.334185108 +0000 UTC m=+31.026445365 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls") pod "dns-default-wq8t9" (UID: "cee8ca5c-ba97-47b5-b22f-2e161bb93fdc") : secret "dns-default-metrics-tls" not found
Oct 30 00:41:17.334266 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:17.334251    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle podName:d95535e9-e677-463a-a662-7e8045c71d9b nodeName:}" failed. No retries permitted until 2026-10-30 00:41:18.334241597 +0000 UTC m=+31.026501858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle") pod "router-default-94c9fcc47-fdtjt" (UID: "d95535e9-e677-463a-a662-7e8045c71d9b") : configmap references non-existent config key: service-ca.crt
Oct 30 00:41:17.334412 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:17.334291    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-tls podName:794aa402-53bf-43fc-b21b-a176bbbcd132 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:18.334280931 +0000 UTC m=+31.026541195 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gh5xc" (UID: "794aa402-53bf-43fc-b21b-a176bbbcd132") : secret "samples-operator-tls" not found
Oct 30 00:41:17.347530 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:17.347324    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jk65g"]
Oct 30 00:41:17.353443 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:41:17.353414    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4dbba6_28af_46b3_ac90_37ce437fd318.slice/crio-361e853b438a0235d45cedd91283a0588a709a51d75f46c2bf559651514d776b WatchSource:0}: Error finding container 361e853b438a0235d45cedd91283a0588a709a51d75f46c2bf559651514d776b: Status 404 returned error can't find the container with id 361e853b438a0235d45cedd91283a0588a709a51d75f46c2bf559651514d776b
Oct 30 00:41:17.383035 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:17.383009    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nkb8p"]
Oct 30 00:41:17.395627 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:17.395576    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xxvch"]
Oct 30 00:41:17.397161 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:17.397139    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-9zmkk"]
Oct 30 00:41:17.401721 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:17.401052    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs"]
Oct 30 00:41:17.410388 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:17.410367    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hf7tk"]
Oct 30 00:41:18.181830 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:18.181726    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nkb8p" event={"ID":"e05b9b53-5f5b-402e-8599-ce61481346e6","Type":"ContainerStarted","Data":"3343d6e6f697450771e01c175f1ff91f886fdd06afd1f8079dfff032504d5282"}
Oct 30 00:41:18.184940 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:18.184850    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hf7tk" event={"ID":"a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55","Type":"ContainerStarted","Data":"2d93152dd6b759165b13a9074c7ea4ba5e69985e2eb6f607478de2cb748f749d"}
Oct 30 00:41:18.190545 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:18.189804    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6q8k5" event={"ID":"4509811e-30a0-4c8b-b21e-58826bd1f6c6","Type":"ContainerStarted","Data":"ea528365676f321b33f601c7fe4325179810b5e2f763701e4f9fa496eea222c3"}
Oct 30 00:41:18.190545 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:18.189850    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6q8k5" event={"ID":"4509811e-30a0-4c8b-b21e-58826bd1f6c6","Type":"ContainerStarted","Data":"53018d64b73fdb4a068c8ca329984997d649894a8bd29be62d2a0095309671b4"}
Oct 30 00:41:18.192496 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:18.192421    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s" event={"ID":"59e5f12b-4a3e-449b-b285-17747780e1cf","Type":"ContainerStarted","Data":"60513a52513278b2ffcb4c04eeda28a50d46522ccefecac86d43bf4faa41f6ee"}
Oct 30 00:41:18.201098 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:18.200956    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9zmkk" event={"ID":"76b2bcb8-636c-46b5-8dfb-95831470349e","Type":"ContainerStarted","Data":"488f9269c3c668a4e1d441fe21955db3244c963444ad131d77de0c1a82ae847d"}
Oct 30 00:41:18.203584 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:18.203531    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g" event={"ID":"8e4dbba6-28af-46b3-ac90-37ce437fd318","Type":"ContainerStarted","Data":"361e853b438a0235d45cedd91283a0588a709a51d75f46c2bf559651514d776b"}
Oct 30 00:41:18.213046 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:18.212565    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pq5xl" event={"ID":"fc3514a7-8b72-40ff-b45a-1c2005b59360","Type":"ContainerStarted","Data":"7cd95ded23e5841f2fc17cc1b310447da326169b51de9ecedd9dbacab2ea452f"}
Oct 30 00:41:18.213046 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:18.212611    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pq5xl" event={"ID":"fc3514a7-8b72-40ff-b45a-1c2005b59360","Type":"ContainerStarted","Data":"a4be861396077b7336d341d30d02aa1c3b1d68b59eabd0533b5a32bf58a784fc"}
Oct 30 00:41:18.215985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:18.215881    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xxvch" event={"ID":"0142f4ce-76c5-4ca1-ac00-c2c334696905","Type":"ContainerStarted","Data":"5dcb777893bb0ed143391b3d0b0eb4304f96ecbef22135c33ea430db6e964cf0"}
Oct 30 00:41:18.223872 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:18.223801    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs" event={"ID":"433f04c0-a9ea-45c8-a2b7-245980607391","Type":"ContainerStarted","Data":"ff40dff7498f99566d9b280ec92caa89e34a8eba8e4a13a0258b6df543ba8747"}
Oct 30 00:41:18.231883 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:18.231352    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-dtn28" event={"ID":"a2dadf26-b255-4281-91f4-98536d370d21","Type":"ContainerStarted","Data":"57cb44b33c33387cb8c1c0e836cefac71769cd0a56185536621a1ab326fe8a6c"}
Oct 30 00:41:18.231883 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:18.231392    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-dtn28" event={"ID":"a2dadf26-b255-4281-91f4-98536d370d21","Type":"ContainerStarted","Data":"3cc619806395200599c333ff4d975468fa2183cea96e8de9ac4cdbc083843bb3"}
Oct 30 00:41:18.235359 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:18.235057    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6q8k5" podStartSLOduration=428.235042763 podStartE2EDuration="7m8.235042763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:18.233243049 +0000 UTC m=+30.925503329" watchObservedRunningTime="2026-10-30 00:41:18.235042763 +0000 UTC m=+30.927303043"
Oct 30 00:41:18.278244 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:18.277194    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-pq5xl" podStartSLOduration=428.277180905 podStartE2EDuration="7m8.277180905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:18.276545908 +0000 UTC m=+30.968806188" watchObservedRunningTime="2026-10-30 00:41:18.277180905 +0000 UTC m=+30.969441184"
Oct 30 00:41:18.348536 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:18.346479    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:18.348536 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:18.346531    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lfnrl\" (UID: \"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:18.348536 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:18.346569    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j7px8\" (UID: \"6a145873-a56a-4fe8-bdb5-dd0e6837b948\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j7px8"
Oct 30 00:41:18.348536 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:18.346690    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:18.348536 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:18.346736    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gh5xc\" (UID: \"794aa402-53bf-43fc-b21b-a176bbbcd132\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"
Oct 30 00:41:18.348536 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:18.346762    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:18.348536 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:18.346801    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:18.348536 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:18.346830    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:18.348536 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:18.347196    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:41:18.348536 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:18.347219    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-654dd5d568-jdc79: secret "image-registry-tls" not found
Oct 30 00:41:18.348536 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:18.347277    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls podName:c48ee8c6-60b2-4229-a013-4199c44813e4 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:20.347258755 +0000 UTC m=+33.039519027 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls") pod "image-registry-654dd5d568-jdc79" (UID: "c48ee8c6-60b2-4229-a013-4199c44813e4") : secret "image-registry-tls" not found
Oct 30 00:41:18.348536 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:18.347701    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:41:18.348536 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:18.347753    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls podName:cee8ca5c-ba97-47b5-b22f-2e161bb93fdc nodeName:}" failed. No retries permitted until 2026-10-30 00:41:20.347735453 +0000 UTC m=+33.039995716 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls") pod "dns-default-wq8t9" (UID: "cee8ca5c-ba97-47b5-b22f-2e161bb93fdc") : secret "dns-default-metrics-tls" not found
Oct 30 00:41:18.348536 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:18.347757    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 00:41:18.348536 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:18.347805    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-tls podName:794aa402-53bf-43fc-b21b-a176bbbcd132 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:20.347790986 +0000 UTC m=+33.040051260 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gh5xc" (UID: "794aa402-53bf-43fc-b21b-a176bbbcd132") : secret "samples-operator-tls" not found
Oct 30 00:41:18.348536 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:18.347810    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 00:41:18.349422 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:18.347846    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls podName:2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc nodeName:}" failed. No retries permitted until 2026-10-30 00:41:20.347834771 +0000 UTC m=+33.040095028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lfnrl" (UID: "2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc") : secret "cluster-monitoring-operator-tls" not found
Oct 30 00:41:18.349422 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:18.347882    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle podName:d95535e9-e677-463a-a662-7e8045c71d9b nodeName:}" failed. No retries permitted until 2026-10-30 00:41:20.347871422 +0000 UTC m=+33.040131701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle") pod "router-default-94c9fcc47-fdtjt" (UID: "d95535e9-e677-463a-a662-7e8045c71d9b") : configmap references non-existent config key: service-ca.crt
Oct 30 00:41:18.349422 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:18.347903    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 00:41:18.349422 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:18.347934    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert podName:6a145873-a56a-4fe8-bdb5-dd0e6837b948 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:20.347923841 +0000 UTC m=+33.040184101 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j7px8" (UID: "6a145873-a56a-4fe8-bdb5-dd0e6837b948") : secret "networking-console-plugin-cert" not found
Oct 30 00:41:18.349422 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:18.347945    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:41:18.349422 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:18.347955    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f45784fb5-r5j44: secret "image-registry-tls" not found
Oct 30 00:41:18.349422 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:18.347983    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls podName:564f9ba3-d19e-4cac-9021-81939eb8e631 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:20.347973863 +0000 UTC m=+33.040234126 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls") pod "image-registry-5f45784fb5-r5j44" (UID: "564f9ba3-d19e-4cac-9021-81939eb8e631") : secret "image-registry-tls" not found
Oct 30 00:41:18.349422 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:18.348036    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 00:41:18.349422 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:18.348064    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs podName:d95535e9-e677-463a-a662-7e8045c71d9b nodeName:}" failed. No retries permitted until 2026-10-30 00:41:20.348055153 +0000 UTC m=+33.040315437 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs") pod "router-default-94c9fcc47-fdtjt" (UID: "d95535e9-e677-463a-a662-7e8045c71d9b") : secret "router-metrics-certs-default" not found
Oct 30 00:41:20.241951 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:20.241196    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nkb8p" event={"ID":"e05b9b53-5f5b-402e-8599-ce61481346e6","Type":"ContainerStarted","Data":"2ed2c3e563d6563c7c37f0739b39949fd2003cb7ea1964953088eaaada279c6a"}
Oct 30 00:41:20.288156 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:20.287649    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-dtn28" podStartSLOduration=430.287633205 podStartE2EDuration="7m10.287633205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:18.321963142 +0000 UTC m=+31.014223422" watchObservedRunningTime="2026-10-30 00:41:20.287633205 +0000 UTC m=+32.979893486"
Oct 30 00:41:20.371757 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:20.371332    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:41:20.371757 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:20.371406    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls podName:cee8ca5c-ba97-47b5-b22f-2e161bb93fdc nodeName:}" failed. No retries permitted until 2026-10-30 00:41:24.371387954 +0000 UTC m=+37.063648227 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls") pod "dns-default-wq8t9" (UID: "cee8ca5c-ba97-47b5-b22f-2e161bb93fdc") : secret "dns-default-metrics-tls" not found
Oct 30 00:41:20.371757 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:20.371724    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:20.372030 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:20.371811    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gh5xc\" (UID: \"794aa402-53bf-43fc-b21b-a176bbbcd132\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"
Oct 30 00:41:20.372030 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:20.371853    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:20.372030 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:20.371898    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:20.372030 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:20.371927    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:20.372030 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:20.371994    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:20.372030 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:20.372020    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lfnrl\" (UID: \"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:20.372354 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:20.372056    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j7px8\" (UID: \"6a145873-a56a-4fe8-bdb5-dd0e6837b948\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j7px8"
Oct 30 00:41:20.372354 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:20.372212    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 00:41:20.372354 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:20.372265    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert podName:6a145873-a56a-4fe8-bdb5-dd0e6837b948 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:24.372248945 +0000 UTC m=+37.064509216 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j7px8" (UID: "6a145873-a56a-4fe8-bdb5-dd0e6837b948") : secret "networking-console-plugin-cert" not found
Oct 30 00:41:20.372354 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:20.372325    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 00:41:20.372544 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:20.372360    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-tls podName:794aa402-53bf-43fc-b21b-a176bbbcd132 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:24.372347882 +0000 UTC m=+37.064608155 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gh5xc" (UID: "794aa402-53bf-43fc-b21b-a176bbbcd132") : secret "samples-operator-tls" not found
Oct 30 00:41:20.372544 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:20.372434    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle podName:d95535e9-e677-463a-a662-7e8045c71d9b nodeName:}" failed. No retries permitted until 2026-10-30 00:41:24.372421671 +0000 UTC m=+37.064681956 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle") pod "router-default-94c9fcc47-fdtjt" (UID: "d95535e9-e677-463a-a662-7e8045c71d9b") : configmap references non-existent config key: service-ca.crt
Oct 30 00:41:20.372544 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:20.372499    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:41:20.372544 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:20.372511    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f45784fb5-r5j44: secret "image-registry-tls" not found
Oct 30 00:41:20.372544 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:20.372540    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls podName:564f9ba3-d19e-4cac-9021-81939eb8e631 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:24.372530651 +0000 UTC m=+37.064790911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls") pod "image-registry-5f45784fb5-r5j44" (UID: "564f9ba3-d19e-4cac-9021-81939eb8e631") : secret "image-registry-tls" not found
Oct 30 00:41:20.372790 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:20.372590    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 00:41:20.372790 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:20.372617    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs podName:d95535e9-e677-463a-a662-7e8045c71d9b nodeName:}" failed. No retries permitted until 2026-10-30 00:41:24.372609098 +0000 UTC m=+37.064869357 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs") pod "router-default-94c9fcc47-fdtjt" (UID: "d95535e9-e677-463a-a662-7e8045c71d9b") : secret "router-metrics-certs-default" not found
Oct 30 00:41:20.372790 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:20.372661    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:41:20.372790 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:20.372670    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-654dd5d568-jdc79: secret "image-registry-tls" not found
Oct 30 00:41:20.372790 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:20.372695    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls podName:c48ee8c6-60b2-4229-a013-4199c44813e4 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:24.372688988 +0000 UTC m=+37.064949257 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls") pod "image-registry-654dd5d568-jdc79" (UID: "c48ee8c6-60b2-4229-a013-4199c44813e4") : secret "image-registry-tls" not found
Oct 30 00:41:20.372790 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:20.372731    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 00:41:20.372790 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:20.372748    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls podName:2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc nodeName:}" failed. No retries permitted until 2026-10-30 00:41:24.372743355 +0000 UTC m=+37.065003611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lfnrl" (UID: "2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc") : secret "cluster-monitoring-operator-tls" not found
Oct 30 00:41:21.586238 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:21.586165    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbef968a-7aa4-4e23-9278-3706e60d8c0b-original-pull-secret\") pod \"global-pull-secret-syncer-wk65n\" (UID: \"bbef968a-7aa4-4e23-9278-3706e60d8c0b\") " pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:41:21.586688 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:21.586253    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs\") pod \"network-metrics-daemon-b4686\" (UID: \"0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493\") " pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:41:21.586688 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:21.586413    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 00:41:21.586688 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:21.586482    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs podName:0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:53.586463912 +0000 UTC m=+66.278724180 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs") pod "network-metrics-daemon-b4686" (UID: "0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493") : secret "metrics-daemon-secret" not found
Oct 30 00:41:21.590598 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:21.590567    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbef968a-7aa4-4e23-9278-3706e60d8c0b-original-pull-secret\") pod \"global-pull-secret-syncer-wk65n\" (UID: \"bbef968a-7aa4-4e23-9278-3706e60d8c0b\") " pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:41:21.650312 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:21.650198    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:41:21.687460 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:21.687425    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vvj2\" (UniqueName: \"kubernetes.io/projected/204212cc-1c4c-4bfb-8b41-03a98042166c-kube-api-access-7vvj2\") pod \"network-check-target-jh728\" (UID: \"204212cc-1c4c-4bfb-8b41-03a98042166c\") " pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:41:21.690622 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:21.690600    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vvj2\" (UniqueName: \"kubernetes.io/projected/204212cc-1c4c-4bfb-8b41-03a98042166c-kube-api-access-7vvj2\") pod \"network-check-target-jh728\" (UID: \"204212cc-1c4c-4bfb-8b41-03a98042166c\") " pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:41:21.981427 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:21.981407    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:41:22.252529 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:22.251982    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xxvch" event={"ID":"0142f4ce-76c5-4ca1-ac00-c2c334696905","Type":"ContainerStarted","Data":"9e812370545aa77a80f3b5432fcab16e59428b16387c8758193886d0053838fd"}
Oct 30 00:41:22.253985 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:22.253955    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs" event={"ID":"433f04c0-a9ea-45c8-a2b7-245980607391","Type":"ContainerStarted","Data":"3b8903639645aeb40bd7c4b0c07088c77293e2bd82c74f2c9159990d372cd7ee"}
Oct 30 00:41:22.263138 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:22.263110    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s" event={"ID":"59e5f12b-4a3e-449b-b285-17747780e1cf","Type":"ContainerStarted","Data":"7bca7e28adb91f11f321ad0f34a0d5c247b3030455427436d7a921d2a064af19"}
Oct 30 00:41:22.264818 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:22.264793    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g" event={"ID":"8e4dbba6-28af-46b3-ac90-37ce437fd318","Type":"ContainerStarted","Data":"ccd60a585e751f1bae6f3e97df485f90d080aaeb253076ad8ed0ead47823216b"}
Oct 30 00:41:22.265331 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:22.265314    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:22.266540 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:22.266506    2582 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-jk65g container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.11:8443/readyz\": dial tcp 10.132.0.11:8443: connect: connection refused" start-of-body=
Oct 30 00:41:22.266632 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:22.266599    2582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g" podUID="8e4dbba6-28af-46b3-ac90-37ce437fd318" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.11:8443/readyz\": dial tcp 10.132.0.11:8443: connect: connection refused"
Oct 30 00:41:22.282814 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:22.282777    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nkb8p" podStartSLOduration=402.847212877 podStartE2EDuration="6m45.282766915s" totalImagesPullingTime="2.435554038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:20.287851172 +0000 UTC m=+32.980111478" watchObservedRunningTime="2026-10-30 00:41:22.282766915 +0000 UTC m=+34.975027194"
Oct 30 00:41:22.283531 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:22.283515    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wk65n"]
Oct 30 00:41:22.298240 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:22.298189    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-xxvch" podStartSLOduration=400.733722882 podStartE2EDuration="6m45.29817401s" totalImagesPullingTime="4.564451128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:22.297217936 +0000 UTC m=+34.989478215" watchObservedRunningTime="2026-10-30 00:41:22.29817401 +0000 UTC m=+34.990434300"
Oct 30 00:41:22.344425 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:22.343839    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jh728"]
Oct 30 00:41:22.346926 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:22.346873    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g" podStartSLOduration=400.730697543 podStartE2EDuration="6m45.346856359s" totalImagesPullingTime="4.616158816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:22.34594849 +0000 UTC m=+35.038208784" watchObservedRunningTime="2026-10-30 00:41:22.346856359 +0000 UTC m=+35.039116638"
Oct 30 00:41:22.390590 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:22.390535    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs" podStartSLOduration=400.832421084 podStartE2EDuration="6m45.390516648s" totalImagesPullingTime="4.558095564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:22.389874175 +0000 UTC m=+35.082134454" watchObservedRunningTime="2026-10-30 00:41:22.390516648 +0000 UTC m=+35.082776928"
Oct 30 00:41:22.391321 ip-10-0-134-210 kubenswrapper[2582]: W1030 00:41:22.390562    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204212cc_1c4c_4bfb_8b41_03a98042166c.slice/crio-0d56c1057633e403e87c7a5c42194323a27dbf826d9452b3a35c0ed5b3382be6 WatchSource:0}: Error finding container 0d56c1057633e403e87c7a5c42194323a27dbf826d9452b3a35c0ed5b3382be6: Status 404 returned error can't find the container with id 0d56c1057633e403e87c7a5c42194323a27dbf826d9452b3a35c0ed5b3382be6
Oct 30 00:41:22.435367 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:22.435309    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s" podStartSLOduration=400.751009537 podStartE2EDuration="6m45.435289486s" totalImagesPullingTime="4.684279949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:22.434159483 +0000 UTC m=+35.126419753" watchObservedRunningTime="2026-10-30 00:41:22.435289486 +0000 UTC m=+35.127549767"
Oct 30 00:41:22.929971 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:22.929946    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f9vrm"]
Oct 30 00:41:22.953858 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:22.953800    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:22.956047 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:22.956016    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f9vrm"]
Oct 30 00:41:23.009443 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.009415    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:23.009568 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.009452    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a50fdaf-7855-4033-96c8-6cb23212a986-data-volume\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:23.009568 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.009491    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxvpz\" (UniqueName: \"kubernetes.io/projected/0a50fdaf-7855-4033-96c8-6cb23212a986-kube-api-access-pxvpz\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:23.009568 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.009510    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a50fdaf-7855-4033-96c8-6cb23212a986-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:23.009683 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.009593    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a50fdaf-7855-4033-96c8-6cb23212a986-crio-socket\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:23.067919 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.067888    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 00:41:23.068083 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.067954    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 00:41:23.068083 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.067954    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qs9gb\""
Oct 30 00:41:23.111182 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.110987    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:23.111182 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.111029    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a50fdaf-7855-4033-96c8-6cb23212a986-data-volume\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:23.111182 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.111104    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxvpz\" (UniqueName: \"kubernetes.io/projected/0a50fdaf-7855-4033-96c8-6cb23212a986-kube-api-access-pxvpz\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:23.111182 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.111132    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a50fdaf-7855-4033-96c8-6cb23212a986-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:23.111182 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:23.111167    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:23.111530 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:23.111227    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls podName:0a50fdaf-7855-4033-96c8-6cb23212a986 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:23.611209239 +0000 UTC m=+36.303469501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f9vrm" (UID: "0a50fdaf-7855-4033-96c8-6cb23212a986") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:23.111530 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.111438    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a50fdaf-7855-4033-96c8-6cb23212a986-data-volume\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:23.111530 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:23.111456    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:41:23.111530 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:23.111512    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a50fdaf-7855-4033-96c8-6cb23212a986-kube-rbac-proxy-cm podName:0a50fdaf-7855-4033-96c8-6cb23212a986 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:23.611495998 +0000 UTC m=+36.303756258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a50fdaf-7855-4033-96c8-6cb23212a986-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f9vrm" (UID: "0a50fdaf-7855-4033-96c8-6cb23212a986") : configmap "kube-rbac-proxy" not found
Oct 30 00:41:23.111792 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.111762    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a50fdaf-7855-4033-96c8-6cb23212a986-crio-socket\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:23.111909 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.111877    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a50fdaf-7855-4033-96c8-6cb23212a986-crio-socket\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:23.141469 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.141438    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxvpz\" (UniqueName: \"kubernetes.io/projected/0a50fdaf-7855-4033-96c8-6cb23212a986-kube-api-access-pxvpz\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:23.271902 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.271766    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hf7tk" event={"ID":"a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55","Type":"ContainerStarted","Data":"fad5e678e221cb50a9040a0a09145bd3983e02fc88ec5d65e796fa4ed4fe6c76"}
Oct 30 00:41:23.279023 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.278965    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jh728" event={"ID":"204212cc-1c4c-4bfb-8b41-03a98042166c","Type":"ContainerStarted","Data":"175d95cc9afcd7273fce165915d8cd2ab30f046fb800d474aa912362e97e9681"}
Oct 30 00:41:23.279236 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.279050    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jh728" event={"ID":"204212cc-1c4c-4bfb-8b41-03a98042166c","Type":"ContainerStarted","Data":"0d56c1057633e403e87c7a5c42194323a27dbf826d9452b3a35c0ed5b3382be6"}
Oct 30 00:41:23.279236 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.279092    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:41:23.280534 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.280508    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9zmkk" event={"ID":"76b2bcb8-636c-46b5-8dfb-95831470349e","Type":"ContainerStarted","Data":"86714acc9ad1c810fa4ceb981820e9c08fe0640dc1d8408aa2f25b70a030b0d4"}
Oct 30 00:41:23.282335 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.282311    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/0.log"
Oct 30 00:41:23.282514 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.282355    2582 generic.go:356] "Generic (PLEG): container finished" podID="8e4dbba6-28af-46b3-ac90-37ce437fd318" containerID="ccd60a585e751f1bae6f3e97df485f90d080aaeb253076ad8ed0ead47823216b" exitCode=255
Oct 30 00:41:23.282622 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.282595    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g" event={"ID":"8e4dbba6-28af-46b3-ac90-37ce437fd318","Type":"ContainerDied","Data":"ccd60a585e751f1bae6f3e97df485f90d080aaeb253076ad8ed0ead47823216b"}
Oct 30 00:41:23.282711 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.282693    2582 scope.go:122] "RemoveContainer" containerID="ccd60a585e751f1bae6f3e97df485f90d080aaeb253076ad8ed0ead47823216b"
Oct 30 00:41:23.284804 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.284782    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wk65n" event={"ID":"bbef968a-7aa4-4e23-9278-3706e60d8c0b","Type":"ContainerStarted","Data":"3345ad6714170a3d6ebec355a44fed3839dc923d622aba74687544a5c9cfe9f2"}
Oct 30 00:41:23.319302 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.319250    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hf7tk" podStartSLOduration=2.341538464 podStartE2EDuration="7.319237331s" totalImagesPullingTime="4.977698867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:23.318445222 +0000 UTC m=+36.010705500" watchObservedRunningTime="2026-10-30 00:41:23.319237331 +0000 UTC m=+36.011497610"
Oct 30 00:41:23.365771 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.365716    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jh728" podStartSLOduration=34.836374446 podStartE2EDuration="35.365703122s" totalImagesPullingTime="529.328676ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:23.364830959 +0000 UTC m=+36.057091238" watchObservedRunningTime="2026-10-30 00:41:23.365703122 +0000 UTC m=+36.057963400"
Oct 30 00:41:23.494969 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.492054    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9zmkk" podStartSLOduration=396.176066307 podStartE2EDuration="6m41.492035523s" totalImagesPullingTime="5.315969216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:23.491269366 +0000 UTC m=+36.183529663" watchObservedRunningTime="2026-10-30 00:41:23.492035523 +0000 UTC m=+36.184295803"
Oct 30 00:41:23.618143 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.618102    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a50fdaf-7855-4033-96c8-6cb23212a986-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:23.618294 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:23.618246    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:41:23.618349 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:23.618330    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a50fdaf-7855-4033-96c8-6cb23212a986-kube-rbac-proxy-cm podName:0a50fdaf-7855-4033-96c8-6cb23212a986 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:24.618308717 +0000 UTC m=+37.310568978 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a50fdaf-7855-4033-96c8-6cb23212a986-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f9vrm" (UID: "0a50fdaf-7855-4033-96c8-6cb23212a986") : configmap "kube-rbac-proxy" not found
Oct 30 00:41:23.618407 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:23.618391    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:23.618523 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:23.618510    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:23.618576 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:23.618542    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls podName:0a50fdaf-7855-4033-96c8-6cb23212a986 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:24.618534101 +0000 UTC m=+37.310794359 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f9vrm" (UID: "0a50fdaf-7855-4033-96c8-6cb23212a986") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:24.289160 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:24.289128    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/1.log"
Oct 30 00:41:24.290009 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:24.289985    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/0.log"
Oct 30 00:41:24.290144 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:24.290019    2582 generic.go:356] "Generic (PLEG): container finished" podID="8e4dbba6-28af-46b3-ac90-37ce437fd318" containerID="a31b42b1c0dd5bc05a1170b6f3ab972f96c72f863fca50fdd3997fb3c7fd610b" exitCode=255
Oct 30 00:41:24.290213 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:24.290139    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g" event={"ID":"8e4dbba6-28af-46b3-ac90-37ce437fd318","Type":"ContainerDied","Data":"a31b42b1c0dd5bc05a1170b6f3ab972f96c72f863fca50fdd3997fb3c7fd610b"}
Oct 30 00:41:24.290213 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:24.290192    2582 scope.go:122] "RemoveContainer" containerID="ccd60a585e751f1bae6f3e97df485f90d080aaeb253076ad8ed0ead47823216b"
Oct 30 00:41:24.290579 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:24.290561    2582 scope.go:122] "RemoveContainer" containerID="a31b42b1c0dd5bc05a1170b6f3ab972f96c72f863fca50fdd3997fb3c7fd610b"
Oct 30 00:41:24.290807 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:24.290785    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jk65g_openshift-console-operator(8e4dbba6-28af-46b3-ac90-37ce437fd318)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g" podUID="8e4dbba6-28af-46b3-ac90-37ce437fd318"
Oct 30 00:41:24.425610 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:24.425561    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:24.425802 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:24.425640    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gh5xc\" (UID: \"794aa402-53bf-43fc-b21b-a176bbbcd132\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"
Oct 30 00:41:24.425802 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:24.425691    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:24.425802 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:24.425730    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:41:24.425802 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:24.425761    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:24.425996 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:24.425807    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 00:41:24.425996 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:24.425822    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls podName:cee8ca5c-ba97-47b5-b22f-2e161bb93fdc nodeName:}" failed. No retries permitted until 2026-10-30 00:41:32.425803282 +0000 UTC m=+45.118063559 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls") pod "dns-default-wq8t9" (UID: "cee8ca5c-ba97-47b5-b22f-2e161bb93fdc") : secret "dns-default-metrics-tls" not found
Oct 30 00:41:24.425996 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:24.425850    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:24.425996 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:24.425870    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:41:24.425996 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:24.425876    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-tls podName:794aa402-53bf-43fc-b21b-a176bbbcd132 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:32.425850845 +0000 UTC m=+45.118111126 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gh5xc" (UID: "794aa402-53bf-43fc-b21b-a176bbbcd132") : secret "samples-operator-tls" not found
Oct 30 00:41:24.425996 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:24.425888    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f45784fb5-r5j44: secret "image-registry-tls" not found
Oct 30 00:41:24.425996 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:24.425912    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 00:41:24.425996 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:24.425922    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle podName:d95535e9-e677-463a-a662-7e8045c71d9b nodeName:}" failed. No retries permitted until 2026-10-30 00:41:32.425906638 +0000 UTC m=+45.118166910 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle") pod "router-default-94c9fcc47-fdtjt" (UID: "d95535e9-e677-463a-a662-7e8045c71d9b") : configmap references non-existent config key: service-ca.crt
Oct 30 00:41:24.425996 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:24.425949    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls podName:564f9ba3-d19e-4cac-9021-81939eb8e631 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:32.425938735 +0000 UTC m=+45.118198992 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls") pod "image-registry-5f45784fb5-r5j44" (UID: "564f9ba3-d19e-4cac-9021-81939eb8e631") : secret "image-registry-tls" not found
Oct 30 00:41:24.425996 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:24.425966    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs podName:d95535e9-e677-463a-a662-7e8045c71d9b nodeName:}" failed. No retries permitted until 2026-10-30 00:41:32.425958848 +0000 UTC m=+45.118219106 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs") pod "router-default-94c9fcc47-fdtjt" (UID: "d95535e9-e677-463a-a662-7e8045c71d9b") : secret "router-metrics-certs-default" not found
Oct 30 00:41:24.426364 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:24.426009    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:24.426364 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:24.426030    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lfnrl\" (UID: \"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:24.426364 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:24.426052    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j7px8\" (UID: \"6a145873-a56a-4fe8-bdb5-dd0e6837b948\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j7px8"
Oct 30 00:41:24.426364 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:24.426179    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 00:41:24.426364 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:24.426205    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert podName:6a145873-a56a-4fe8-bdb5-dd0e6837b948 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:32.426198098 +0000 UTC m=+45.118458354 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j7px8" (UID: "6a145873-a56a-4fe8-bdb5-dd0e6837b948") : secret "networking-console-plugin-cert" not found
Oct 30 00:41:24.426364 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:24.426227    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:41:24.426364 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:24.426239    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-654dd5d568-jdc79: secret "image-registry-tls" not found
Oct 30 00:41:24.426364 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:24.426257    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 00:41:24.426364 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:24.426280    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls podName:c48ee8c6-60b2-4229-a013-4199c44813e4 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:32.426269996 +0000 UTC m=+45.118530253 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls") pod "image-registry-654dd5d568-jdc79" (UID: "c48ee8c6-60b2-4229-a013-4199c44813e4") : secret "image-registry-tls" not found
Oct 30 00:41:24.426364 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:24.426292    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls podName:2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc nodeName:}" failed. No retries permitted until 2026-10-30 00:41:32.426286768 +0000 UTC m=+45.118547024 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lfnrl" (UID: "2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc") : secret "cluster-monitoring-operator-tls" not found
Oct 30 00:41:24.627274 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:24.627202    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:24.627474 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:24.627281    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a50fdaf-7855-4033-96c8-6cb23212a986-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:24.627474 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:24.627376    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:24.627474 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:24.627403    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:41:24.627474 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:24.627438    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls podName:0a50fdaf-7855-4033-96c8-6cb23212a986 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:26.627420859 +0000 UTC m=+39.319681120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f9vrm" (UID: "0a50fdaf-7855-4033-96c8-6cb23212a986") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:24.627474 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:24.627465    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a50fdaf-7855-4033-96c8-6cb23212a986-kube-rbac-proxy-cm podName:0a50fdaf-7855-4033-96c8-6cb23212a986 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:26.627448947 +0000 UTC m=+39.319709207 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a50fdaf-7855-4033-96c8-6cb23212a986-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f9vrm" (UID: "0a50fdaf-7855-4033-96c8-6cb23212a986") : configmap "kube-rbac-proxy" not found
Oct 30 00:41:25.293901 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:25.293873    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/1.log"
Oct 30 00:41:25.294304 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:25.294236    2582 scope.go:122] "RemoveContainer" containerID="a31b42b1c0dd5bc05a1170b6f3ab972f96c72f863fca50fdd3997fb3c7fd610b"
Oct 30 00:41:25.294428 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:25.294410    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jk65g_openshift-console-operator(8e4dbba6-28af-46b3-ac90-37ce437fd318)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g" podUID="8e4dbba6-28af-46b3-ac90-37ce437fd318"
Oct 30 00:41:26.646307 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:26.646274    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a50fdaf-7855-4033-96c8-6cb23212a986-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:26.646912 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:26.646418    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:41:26.646912 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:26.646492    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a50fdaf-7855-4033-96c8-6cb23212a986-kube-rbac-proxy-cm podName:0a50fdaf-7855-4033-96c8-6cb23212a986 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:30.64647206 +0000 UTC m=+43.338732337 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a50fdaf-7855-4033-96c8-6cb23212a986-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f9vrm" (UID: "0a50fdaf-7855-4033-96c8-6cb23212a986") : configmap "kube-rbac-proxy" not found
Oct 30 00:41:26.646912 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:26.646517    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:26.646912 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:26.646685    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:26.646912 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:26.646722    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls podName:0a50fdaf-7855-4033-96c8-6cb23212a986 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:30.646713074 +0000 UTC m=+43.338973345 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f9vrm" (UID: "0a50fdaf-7855-4033-96c8-6cb23212a986") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:26.972218 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:26.972184    2582 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:26.972552 ip-10-0-134-210 kubenswrapper[2582]: I1030 00:41:26.972539    2582 scope.go:122] "RemoveContainer" containerID="a31b42b1c0dd5bc05a1170b6f3ab972f96c72f863fca50fdd3997fb3c7fd610b"
Oct 30 00:41:26.972708 ip-10-0-134-210 kubenswrapper[2582]: E1030 00:41:26.972692    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jk65g_openshift-console-operator(8e4dbba6-28af-46b3-ac90-37ce437fd318)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g" podUID="8e4dbba6-28af-46b3-ac90-37ce437fd318"
Oct 30 00:41:27.589408 ip-10-0-134-210 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 00:41:27.599483 ip-10-0-134-210 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 00:41:27.599643 ip-10-0-134-210 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 00:41:27.599816 ip-10-0-134-210 systemd[1]: kubelet.service: Consumed 2.906s CPU time.
Oct 30 00:41:27.607688 ip-10-0-134-210 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 00:41:27.700414 ip-10-0-134-210 kubenswrapper[4864]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:41:27.700414 ip-10-0-134-210 kubenswrapper[4864]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 00:41:27.700414 ip-10-0-134-210 kubenswrapper[4864]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:41:27.700414 ip-10-0-134-210 kubenswrapper[4864]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:41:27.702541 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702455    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:41:27.702541 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702539    4864 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:41:27.702541 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702543    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:41:27.702639 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702547    4864 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:41:27.702639 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702550    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:41:27.702639 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702553    4864 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:41:27.702639 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702556    4864 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:41:27.702639 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702564    4864 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:41:27.702639 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702569    4864 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:41:27.702639 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702574    4864 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:41:27.702639 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702577    4864 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:41:27.702639 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702580    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:41:27.702639 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702583    4864 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:41:27.702639 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702586    4864 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:41:27.702639 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702589    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:41:27.702639 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702593    4864 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:41:27.702639 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702598    4864 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:41:27.702639 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702601    4864 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:41:27.702639 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702604    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:41:27.702639 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702606    4864 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:41:27.702639 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702609    4864 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:41:27.703114 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702612    4864 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:41:27.703114 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702614    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:41:27.703114 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702617    4864 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:41:27.703114 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702620    4864 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:41:27.703114 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702623    4864 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:41:27.703114 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702626    4864 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:41:27.703114 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702629    4864 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:41:27.703114 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702631    4864 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:41:27.703114 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702634    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:41:27.703114 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702637    4864 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:41:27.703114 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702639    4864 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:41:27.703114 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702642    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:41:27.703114 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702645    4864 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:41:27.703114 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702648    4864 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:41:27.703114 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702651    4864 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:41:27.703114 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702654    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:41:27.703114 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702656    4864 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:41:27.703114 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702659    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:41:27.703114 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702662    4864 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:41:27.703114 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702665    4864 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:41:27.703620 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702667    4864 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:41:27.703620 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702670    4864 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:41:27.703620 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702672    4864 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:41:27.703620 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702675    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:41:27.703620 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702678    4864 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:41:27.703620 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702680    4864 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:41:27.703620 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702683    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:41:27.703620 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702685    4864 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:41:27.703620 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702688    4864 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:41:27.703620 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702690    4864 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:41:27.703620 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702693    4864 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:41:27.703620 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702696    4864 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:41:27.703620 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702699    4864 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:41:27.703620 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702701    4864 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:41:27.703620 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702704    4864 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:41:27.703620 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702706    4864 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:41:27.703620 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702709    4864 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:41:27.703620 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702713    4864 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:41:27.703620 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702716    4864 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:41:27.704097 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702718    4864 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:41:27.704097 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702721    4864 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:41:27.704097 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702724    4864 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:41:27.704097 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702726    4864 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:41:27.704097 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702729    4864 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:41:27.704097 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702731    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:41:27.704097 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702735    4864 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:41:27.704097 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702737    4864 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:41:27.704097 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702739    4864 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:41:27.704097 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702742    4864 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:41:27.704097 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702745    4864 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:41:27.704097 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702747    4864 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:41:27.704097 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702750    4864 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:41:27.704097 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702753    4864 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:41:27.704097 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702756    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:41:27.704097 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702759    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:41:27.704097 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702762    4864 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:41:27.704097 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702764    4864 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:41:27.704097 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702767    4864 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:41:27.704097 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702770    4864 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:41:27.704597 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702772    4864 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:41:27.704597 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702775    4864 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:41:27.704597 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702778    4864 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:41:27.704597 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702780    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:41:27.704597 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702783    4864 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:41:27.704597 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702786    4864 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:41:27.704597 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702788    4864 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:41:27.704597 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702791    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:41:27.704597 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702793    4864 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:41:27.704597 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702796    4864 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:41:27.704597 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702799    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:41:27.704597 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702801    4864 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:41:27.704597 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702804    4864 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:41:27.704597 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702807    4864 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:41:27.704597 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702809    4864 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:41:27.704597 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702812    4864 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:41:27.704597 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702815    4864 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:41:27.704597 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702817    4864 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:41:27.704597 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702820    4864 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:41:27.704597 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702823    4864 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:41:27.705143 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702826    4864 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:41:27.705143 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702828    4864 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:41:27.705143 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702831    4864 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:41:27.705143 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702833    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:41:27.705143 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702836    4864 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:41:27.705143 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.702840    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:41:27.705143 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703624    4864 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:41:27.705143 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703631    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:41:27.705143 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703634    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:41:27.705143 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703637    4864 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:41:27.705143 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703640    4864 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:41:27.705143 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703644    4864 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:41:27.705143 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703647    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:41:27.705143 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703650    4864 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:41:27.705143 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703653    4864 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:41:27.705143 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703656    4864 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:41:27.705143 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703658    4864 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:41:27.705143 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703661    4864 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:41:27.705143 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703664    4864 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:41:27.705143 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703667    4864 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:41:27.705653 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703669    4864 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:41:27.705653 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703672    4864 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:41:27.705653 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703674    4864 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:41:27.705653 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703677    4864 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:41:27.705653 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703680    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:41:27.705653 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703682    4864 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:41:27.705653 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703685    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:41:27.705653 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703687    4864 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:41:27.705653 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703690    4864 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:41:27.705653 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703693    4864 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:41:27.705653 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703696    4864 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:41:27.705653 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703698    4864 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:41:27.705653 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703701    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:41:27.705653 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703704    4864 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:41:27.705653 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703706    4864 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:41:27.705653 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703708    4864 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:41:27.705653 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703711    4864 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:41:27.705653 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703714    4864 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:41:27.705653 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703717    4864 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:41:27.706150 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703720    4864 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:41:27.706150 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703723    4864 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:41:27.706150 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703725    4864 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:41:27.706150 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703730    4864 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:41:27.706150 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703738    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:41:27.706150 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703741    4864 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:41:27.706150 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703744    4864 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:41:27.706150 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703747    4864 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:41:27.706150 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703749    4864 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:41:27.706150 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703752    4864 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:41:27.706150 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703755    4864 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:41:27.706150 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703757    4864 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:41:27.706150 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703760    4864 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:41:27.706150 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703763    4864 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:41:27.706150 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703765    4864 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:41:27.706150 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703768    4864 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:41:27.706150 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703770    4864 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:41:27.706150 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703773    4864 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:41:27.706150 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703777    4864 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:41:27.706683 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703780    4864 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:41:27.706683 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703782    4864 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:41:27.706683 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703785    4864 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:41:27.706683 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703788    4864 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:41:27.706683 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703791    4864 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:41:27.706683 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703793    4864 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:41:27.706683 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703796    4864 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:41:27.706683 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703799    4864 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:41:27.706683 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703801    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:41:27.706683 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703804    4864 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:41:27.706683 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703806    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:41:27.706683 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703809    4864 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:41:27.706683 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703812    4864 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:41:27.706683 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703815    4864 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:41:27.706683 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703817    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:41:27.706683 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703820    4864 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:41:27.706683 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703822    4864 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:41:27.706683 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703827    4864 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:41:27.706683 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703829    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:41:27.707194 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703832    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:41:27.707194 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703834    4864 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:41:27.707194 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703837    4864 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:41:27.707194 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703840    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:41:27.707194 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703842    4864 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:41:27.707194 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703845    4864 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:41:27.707194 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703847    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:41:27.707194 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703850    4864 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:41:27.707194 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703852    4864 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:41:27.707194 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703855    4864 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:41:27.707194 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703858    4864 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:41:27.707194 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703860    4864 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:41:27.707194 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703863    4864 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:41:27.707194 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703865    4864 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:41:27.707194 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703868    4864 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:41:27.707194 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703871    4864 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:41:27.707194 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703873    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:41:27.707194 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703876    4864 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:41:27.707194 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703879    4864 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:41:27.707194 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703881    4864 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:41:27.707698 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703884    4864 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:41:27.707698 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703886    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:41:27.707698 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703889    4864 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:41:27.707698 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703892    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:41:27.707698 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703894    4864 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:41:27.707698 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703897    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:41:27.707698 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703899    4864 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:41:27.707698 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703903    4864 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:41:27.707698 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703905    4864 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:41:27.707698 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703908    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:41:27.707698 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703914    4864 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:41:27.707698 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703918    4864 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:41:27.707698 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703921    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:41:27.707698 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703925    4864 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:41:27.707698 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.703929    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:41:27.707698 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704397    4864 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:41:27.707698 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704402    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:41:27.707698 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704405    4864 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:41:27.707698 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704408    4864 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:41:27.708177 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704411    4864 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:41:27.708177 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704413    4864 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:41:27.708177 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704416    4864 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:41:27.708177 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704419    4864 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:41:27.708177 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704421    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:41:27.708177 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704425    4864 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:41:27.708177 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704427    4864 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:41:27.708177 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704430    4864 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:41:27.708177 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704433    4864 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:41:27.708177 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704435    4864 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:41:27.708177 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704438    4864 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:41:27.708177 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704441    4864 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:41:27.708177 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704443    4864 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:41:27.708177 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704446    4864 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:41:27.708177 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704449    4864 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:41:27.708177 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704451    4864 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:41:27.708177 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704454    4864 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:41:27.708177 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704456    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:41:27.708177 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704459    4864 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:41:27.708177 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704462    4864 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:41:27.708691 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704465    4864 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:41:27.708691 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704469    4864 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:41:27.708691 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704473    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:41:27.708691 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704476    4864 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:41:27.708691 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704479    4864 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:41:27.708691 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704481    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:41:27.708691 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704484    4864 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:41:27.708691 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704487    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:41:27.708691 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704489    4864 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:41:27.708691 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704492    4864 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:41:27.708691 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704494    4864 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:41:27.708691 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704497    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:41:27.708691 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704500    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:41:27.708691 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704503    4864 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:41:27.708691 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704505    4864 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:41:27.708691 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704508    4864 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:41:27.708691 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704510    4864 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:41:27.708691 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704513    4864 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:41:27.708691 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704515    4864 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:41:27.709423 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704518    4864 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:41:27.709423 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704520    4864 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:41:27.709423 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704523    4864 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:41:27.709423 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704525    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:41:27.709423 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704528    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:41:27.709423 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704531    4864 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:41:27.709423 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704534    4864 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:41:27.709423 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704536    4864 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:41:27.709423 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704539    4864 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:41:27.709423 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704544    4864 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:41:27.709423 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704547    4864 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:41:27.709423 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704550    4864 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:41:27.709423 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704552    4864 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:41:27.709423 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704555    4864 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:41:27.709423 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704558    4864 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:41:27.709423 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704562    4864 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:41:27.709423 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704566    4864 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:41:27.709423 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704569    4864 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:41:27.709423 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704571    4864 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:41:27.710058 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704574    4864 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:41:27.710058 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704577    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:41:27.710058 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704579    4864 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:41:27.710058 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704582    4864 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:41:27.710058 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704584    4864 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:41:27.710058 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704587    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:41:27.710058 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704590    4864 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:41:27.710058 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704592    4864 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:41:27.710058 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704594    4864 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:41:27.710058 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704597    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:41:27.710058 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704600    4864 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:41:27.710058 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704602    4864 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:41:27.710058 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704605    4864 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:41:27.710058 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704608    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:41:27.710058 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704611    4864 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:41:27.710058 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704614    4864 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:41:27.710058 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704616    4864 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:41:27.710058 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704619    4864 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:41:27.710058 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704622    4864 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:41:27.710058 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704624    4864 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:41:27.710584 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704627    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:41:27.710584 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704630    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:41:27.710584 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704634    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:41:27.710584 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704637    4864 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:41:27.710584 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704639    4864 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:41:27.710584 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704642    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:41:27.710584 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704644    4864 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:41:27.710584 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704647    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:41:27.710584 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704650    4864 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:41:27.710584 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704652    4864 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:41:27.710584 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704655    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:41:27.710584 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704657    4864 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:41:27.710584 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704660    4864 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:41:27.710584 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704663    4864 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:41:27.710584 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704665    4864 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:41:27.710584 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704668    4864 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:41:27.710584 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704670    4864 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:41:27.710584 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704673    4864 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:41:27.710584 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704675    4864 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:41:27.711057 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704678    4864 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:41:27.711057 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704680    4864 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:41:27.711057 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704683    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:41:27.711057 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704685    4864 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:41:27.711057 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.704688    4864 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:41:27.711057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704793    4864 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 00:41:27.711057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704807    4864 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 00:41:27.711057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704814    4864 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 00:41:27.711057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704819    4864 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 00:41:27.711057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704824    4864 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 00:41:27.711057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704827    4864 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 00:41:27.711057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704831    4864 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 00:41:27.711057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704836    4864 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 00:41:27.711057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704839    4864 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 00:41:27.711057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704842    4864 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 00:41:27.711057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704847    4864 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 00:41:27.711057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704851    4864 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 00:41:27.711057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704854    4864 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 00:41:27.711057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704857    4864 flags.go:64] FLAG: --cgroup-root=""
Oct 30 00:41:27.711057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704860    4864 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 00:41:27.711057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704863    4864 flags.go:64] FLAG: --client-ca-file=""
Oct 30 00:41:27.711057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704866    4864 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 00:41:27.711057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704870    4864 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 00:41:27.711657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704874    4864 flags.go:64] FLAG: --cluster-domain=""
Oct 30 00:41:27.711657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704877    4864 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 00:41:27.711657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704880    4864 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 00:41:27.711657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704883    4864 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 00:41:27.711657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704887    4864 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 00:41:27.711657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704891    4864 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 00:41:27.711657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704894    4864 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 00:41:27.711657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704897    4864 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 00:41:27.711657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704900    4864 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 00:41:27.711657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704903    4864 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 00:41:27.711657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704908    4864 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 00:41:27.711657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704911    4864 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 00:41:27.711657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704915    4864 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 00:41:27.711657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704918    4864 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 00:41:27.711657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704922    4864 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 00:41:27.711657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704925    4864 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 00:41:27.711657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704928    4864 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 00:41:27.711657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704931    4864 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 00:41:27.711657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704934    4864 flags.go:64] FLAG: --enable-server="true"
Oct 30 00:41:27.711657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704937    4864 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 00:41:27.711657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704948    4864 flags.go:64] FLAG: --event-burst="100"
Oct 30 00:41:27.711657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704952    4864 flags.go:64] FLAG: --event-qps="50"
Oct 30 00:41:27.711657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704955    4864 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 00:41:27.711657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704958    4864 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 00:41:27.712296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704963    4864 flags.go:64] FLAG: --eviction-hard=""
Oct 30 00:41:27.712296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704971    4864 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 00:41:27.712296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704975    4864 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 00:41:27.712296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704978    4864 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 00:41:27.712296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704981    4864 flags.go:64] FLAG: --eviction-soft=""
Oct 30 00:41:27.712296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704984    4864 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 00:41:27.712296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704987    4864 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 00:41:27.712296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704990    4864 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 00:41:27.712296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704993    4864 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 00:41:27.712296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.704996    4864 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 00:41:27.712296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705000    4864 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 00:41:27.712296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705003    4864 flags.go:64] FLAG: --feature-gates=""
Oct 30 00:41:27.712296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705007    4864 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 00:41:27.712296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705010    4864 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 00:41:27.712296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705013    4864 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 00:41:27.712296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705016    4864 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 00:41:27.712296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705020    4864 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 00:41:27.712296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705023    4864 flags.go:64] FLAG: --help="false"
Oct 30 00:41:27.712296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705027    4864 flags.go:64] FLAG: --hostname-override="ip-10-0-134-210.ec2.internal"
Oct 30 00:41:27.712296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705030    4864 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 00:41:27.712296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705033    4864 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 00:41:27.712296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705036    4864 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 00:41:27.712296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705049    4864 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 00:41:27.712296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705053    4864 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705056    4864 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705059    4864 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705063    4864 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705066    4864 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705081    4864 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705084    4864 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705087    4864 flags.go:64] FLAG: --kube-reserved=""
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705091    4864 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705095    4864 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705098    4864 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705101    4864 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705104    4864 flags.go:64] FLAG: --lock-file=""
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705107    4864 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705110    4864 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705113    4864 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705119    4864 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705122    4864 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705126    4864 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705129    4864 flags.go:64] FLAG: --logging-format="text"
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705132    4864 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705136    4864 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705139    4864 flags.go:64] FLAG: --manifest-url=""
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705142    4864 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705147    4864 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 00:41:27.712910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705150    4864 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705155    4864 flags.go:64] FLAG: --max-pods="110"
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705159    4864 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705162    4864 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705165    4864 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705168    4864 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705172    4864 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705175    4864 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705178    4864 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705184    4864 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705188    4864 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705191    4864 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705194    4864 flags.go:64] FLAG: --pod-cidr=""
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705197    4864 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705200    4864 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705203    4864 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705208    4864 flags.go:64] FLAG: --port="10250"
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705211    4864 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705215    4864 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07159f16fd508fd11"
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705218    4864 flags.go:64] FLAG: --qos-reserved=""
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705221    4864 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705224    4864 flags.go:64] FLAG: --register-node="true"
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705227    4864 flags.go:64] FLAG: --register-with-taints=""
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705231    4864 flags.go:64] FLAG: --registry-burst="10"
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705234    4864 flags.go:64] FLAG: --registry-qps="5"
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705237    4864 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 00:41:27.713523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705240    4864 flags.go:64] FLAG: --reserved-memory=""
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705244    4864 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705248    4864 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705251    4864 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705254    4864 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705256    4864 flags.go:64] FLAG: --runonce="false"
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705260    4864 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705263    4864 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705267    4864 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705270    4864 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705273    4864 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705276    4864 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705279    4864 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705283    4864 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705286    4864 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705289    4864 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705292    4864 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705295    4864 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705298    4864 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705302    4864 flags.go:64] FLAG: --system-cgroups=""
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705304    4864 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705310    4864 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705314    4864 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705317    4864 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705321    4864 flags.go:64] FLAG: --tls-min-version=""
Oct 30 00:41:27.714257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705324    4864 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 00:41:27.714870 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705327    4864 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 00:41:27.714870 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705330    4864 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 00:41:27.714870 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705333    4864 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 00:41:27.714870 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705336    4864 flags.go:64] FLAG: --v="2"
Oct 30 00:41:27.714870 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705341    4864 flags.go:64] FLAG: --version="false"
Oct 30 00:41:27.714870 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705346    4864 flags.go:64] FLAG: --vmodule=""
Oct 30 00:41:27.714870 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705350    4864 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 00:41:27.714870 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705354    4864 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 00:41:27.714870 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705497    4864 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:41:27.714870 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705501    4864 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:41:27.714870 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705505    4864 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:41:27.714870 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705509    4864 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:41:27.714870 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705512    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:41:27.714870 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705515    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:41:27.714870 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705520    4864 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:41:27.714870 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705524    4864 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:41:27.714870 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705527    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:41:27.714870 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705530    4864 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:41:27.714870 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705533    4864 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:41:27.714870 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705536    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:41:27.714870 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705539    4864 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:41:27.715418 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705541    4864 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:41:27.715418 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705544    4864 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:41:27.715418 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705546    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:41:27.715418 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705549    4864 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:41:27.715418 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705552    4864 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:41:27.715418 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705555    4864 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:41:27.715418 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705557    4864 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:41:27.715418 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705561    4864 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:41:27.715418 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705564    4864 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:41:27.715418 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705567    4864 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:41:27.715418 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705569    4864 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:41:27.715418 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705572    4864 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:41:27.715418 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705575    4864 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:41:27.715418 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705578    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:41:27.715418 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705580    4864 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:41:27.715418 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705583    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:41:27.715418 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705587    4864 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:41:27.715418 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705591    4864 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:41:27.715418 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705594    4864 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:41:27.715874 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705598    4864 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:41:27.715874 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705600    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:41:27.715874 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705603    4864 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:41:27.715874 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705606    4864 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:41:27.715874 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705608    4864 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:41:27.715874 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705611    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:41:27.715874 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705616    4864 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:41:27.715874 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705618    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:41:27.715874 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705621    4864 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:41:27.715874 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705624    4864 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:41:27.715874 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705626    4864 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:41:27.715874 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705629    4864 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:41:27.715874 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705631    4864 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:41:27.715874 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705634    4864 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:41:27.715874 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705637    4864 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:41:27.715874 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705639    4864 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:41:27.715874 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705642    4864 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:41:27.715874 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705645    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:41:27.715874 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705647    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:41:27.715874 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705650    4864 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:41:27.716370 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705654    4864 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:41:27.716370 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705656    4864 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:41:27.716370 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705659    4864 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:41:27.716370 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705662    4864 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:41:27.716370 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705665    4864 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:41:27.716370 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705668    4864 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:41:27.716370 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705670    4864 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:41:27.716370 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705673    4864 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:41:27.716370 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705675    4864 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:41:27.716370 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705678    4864 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:41:27.716370 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705681    4864 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:41:27.716370 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705683    4864 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:41:27.716370 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705686    4864 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:41:27.716370 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705689    4864 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:41:27.716370 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705692    4864 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:41:27.716370 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705695    4864 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:41:27.716370 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705697    4864 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:41:27.716370 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705700    4864 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:41:27.716370 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705704    4864 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:41:27.716370 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705707    4864 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:41:27.716862 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705709    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:41:27.716862 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705712    4864 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:41:27.716862 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705715    4864 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:41:27.716862 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705717    4864 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:41:27.716862 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705720    4864 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:41:27.716862 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705723    4864 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:41:27.716862 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705725    4864 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:41:27.716862 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705728    4864 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:41:27.716862 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705730    4864 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:41:27.716862 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705733    4864 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:41:27.716862 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705736    4864 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:41:27.716862 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705739    4864 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:41:27.716862 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705742    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:41:27.716862 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705744    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:41:27.716862 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705747    4864 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:41:27.716862 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705750    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:41:27.716862 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705752    4864 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:41:27.716862 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705755    4864 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:41:27.716862 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705758    4864 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:41:27.716862 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705760    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:41:27.717350 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705763    4864 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:41:27.717350 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705765    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:41:27.717350 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705768    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:41:27.717350 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705771    4864 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:41:27.717350 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705773    4864 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:41:27.717350 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705776    4864 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:41:27.717350 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705779    4864 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:41:27.717350 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705782    4864 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:41:27.717350 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705784    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:41:27.717350 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705787    4864 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:41:27.717350 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705791    4864 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:41:27.717350 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705793    4864 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:41:27.717350 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705796    4864 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:41:27.717350 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705799    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:41:27.717350 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.705824    4864 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:41:27.717350 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705857    4864 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:41:27.717740 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705862    4864 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:41:27.717740 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705865    4864 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:41:27.717740 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705868    4864 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:41:27.717740 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705871    4864 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:41:27.717740 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705874    4864 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:41:27.717740 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705878    4864 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:41:27.717740 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705881    4864 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:41:27.717740 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705884    4864 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:41:27.717740 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705887    4864 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:41:27.717740 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705890    4864 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:41:27.717740 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705893    4864 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:41:27.717740 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705895    4864 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:41:27.717740 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705898    4864 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:41:27.717740 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705901    4864 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:41:27.717740 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705904    4864 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:41:27.717740 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705907    4864 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:41:27.717740 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705909    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:41:27.717740 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705912    4864 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:41:27.717740 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705914    4864 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:41:27.718229 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705917    4864 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:41:27.718229 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705919    4864 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:41:27.718229 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705922    4864 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:41:27.718229 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705925    4864 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:41:27.718229 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705928    4864 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:41:27.718229 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705931    4864 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:41:27.718229 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705933    4864 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:41:27.718229 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705936    4864 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:41:27.718229 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705940    4864 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:41:27.718229 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705942    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:41:27.718229 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705945    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:41:27.718229 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705948    4864 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:41:27.718229 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705950    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:41:27.718229 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705953    4864 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:41:27.718229 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705956    4864 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:41:27.718229 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705958    4864 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:41:27.718229 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705961    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:41:27.718229 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705964    4864 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:41:27.718229 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705966    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:41:27.718692 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705969    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:41:27.718692 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705972    4864 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:41:27.718692 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705974    4864 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:41:27.718692 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705977    4864 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:41:27.718692 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705980    4864 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:41:27.718692 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705982    4864 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:41:27.718692 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705985    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:41:27.718692 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705987    4864 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:41:27.718692 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705990    4864 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:41:27.718692 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705992    4864 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:41:27.718692 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705995    4864 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:41:27.718692 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.705998    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:41:27.718692 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706000    4864 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:41:27.718692 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706003    4864 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:41:27.718692 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706006    4864 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:41:27.718692 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706008    4864 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:41:27.718692 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706011    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:41:27.718692 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706013    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:41:27.718692 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706018    4864 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:41:27.719175 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706022    4864 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:41:27.719175 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706025    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:41:27.719175 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706029    4864 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:41:27.719175 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706032    4864 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:41:27.719175 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706035    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:41:27.719175 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706038    4864 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:41:27.719175 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706040    4864 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:41:27.719175 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706043    4864 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:41:27.719175 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706045    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:41:27.719175 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706048    4864 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:41:27.719175 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706051    4864 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:41:27.719175 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706054    4864 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:41:27.719175 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706057    4864 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:41:27.719175 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706059    4864 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:41:27.719175 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706062    4864 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:41:27.719175 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706064    4864 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:41:27.719175 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706080    4864 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:41:27.719175 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706083    4864 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:41:27.719175 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706086    4864 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:41:27.719723 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706089    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:41:27.719723 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706091    4864 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:41:27.719723 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706094    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:41:27.719723 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706098    4864 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:41:27.719723 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706102    4864 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:41:27.719723 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706105    4864 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:41:27.719723 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706108    4864 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:41:27.719723 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706112    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:41:27.719723 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706115    4864 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:41:27.719723 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706118    4864 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:41:27.719723 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706120    4864 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:41:27.719723 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706123    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:41:27.719723 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706126    4864 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:41:27.719723 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706129    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:41:27.719723 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706131    4864 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:41:27.719723 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706136    4864 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:41:27.719723 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706139    4864 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:41:27.719723 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706141    4864 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:41:27.719723 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706144    4864 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:41:27.720293 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706147    4864 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:41:27.720293 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706150    4864 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:41:27.720293 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706152    4864 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:41:27.720293 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706164    4864 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:41:27.720293 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706167    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:41:27.720293 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706170    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:41:27.720293 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706173    4864 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:41:27.720293 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706176    4864 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:41:27.720293 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706178    4864 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:41:27.720293 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.706181    4864 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:41:27.720293 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.711156    4864 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 00:41:27.720293 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.711172    4864 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 00:41:27.720293 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.711195    4864 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 00:41:27.720293 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.711201    4864 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 00:41:27.720293 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711263    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:41:27.720293 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711268    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:41:27.720293 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711271    4864 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:41:27.720293 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711274    4864 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:41:27.720293 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711276    4864 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:41:27.720293 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711279    4864 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:41:27.720970 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711283    4864 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:41:27.720970 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711285    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:41:27.720970 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711288    4864 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:41:27.720970 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711291    4864 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:41:27.720970 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711294    4864 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:41:27.720970 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711298    4864 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:41:27.720970 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711302    4864 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:41:27.720970 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711305    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:41:27.720970 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711313    4864 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:41:27.720970 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711317    4864 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:41:27.720970 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711320    4864 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:41:27.720970 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711322    4864 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:41:27.720970 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711325    4864 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:41:27.720970 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711328    4864 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:41:27.720970 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711330    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:41:27.720970 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711333    4864 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:41:27.720970 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711335    4864 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:41:27.720970 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711338    4864 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:41:27.720970 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711341    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:41:27.721609 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711343    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:41:27.721609 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711346    4864 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:41:27.721609 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711348    4864 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:41:27.721609 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711351    4864 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:41:27.721609 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711354    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:41:27.721609 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711356    4864 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:41:27.721609 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711359    4864 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:41:27.721609 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711362    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:41:27.721609 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711365    4864 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:41:27.721609 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711368    4864 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:41:27.721609 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711370    4864 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:41:27.721609 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711373    4864 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:41:27.721609 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711375    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:41:27.721609 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711378    4864 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:41:27.721609 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711381    4864 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:41:27.721609 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711383    4864 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:41:27.721609 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711386    4864 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:41:27.721609 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711388    4864 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:41:27.721609 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711391    4864 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:41:27.721609 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711393    4864 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:41:27.722107 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711396    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:41:27.722107 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711402    4864 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:41:27.722107 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711405    4864 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:41:27.722107 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711408    4864 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:41:27.722107 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711411    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:41:27.722107 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711413    4864 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:41:27.722107 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711416    4864 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:41:27.722107 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711419    4864 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:41:27.722107 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711422    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:41:27.722107 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711424    4864 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:41:27.722107 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711427    4864 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:41:27.722107 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711429    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:41:27.722107 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711432    4864 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:41:27.722107 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711434    4864 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:41:27.722107 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711437    4864 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:41:27.722107 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711440    4864 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:41:27.722107 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711443    4864 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:41:27.722107 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711447    4864 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:41:27.722616 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711451    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:41:27.722616 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711454    4864 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:41:27.722616 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711457    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:41:27.722616 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711460    4864 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:41:27.722616 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711462    4864 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:41:27.722616 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711465    4864 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:41:27.722616 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711468    4864 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:41:27.722616 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711470    4864 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:41:27.722616 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711473    4864 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:41:27.722616 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711476    4864 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:41:27.722616 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711479    4864 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:41:27.722616 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711481    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:41:27.722616 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711484    4864 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:41:27.722616 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711486    4864 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:41:27.722616 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711489    4864 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:41:27.722616 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711492    4864 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:41:27.722616 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711495    4864 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:41:27.722616 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711498    4864 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:41:27.722616 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711501    4864 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:41:27.722616 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711503    4864 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:41:27.723121 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711506    4864 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:41:27.723121 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711508    4864 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:41:27.723121 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711511    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:41:27.723121 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711513    4864 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:41:27.723121 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711516    4864 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:41:27.723121 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711518    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:41:27.723121 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711521    4864 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:41:27.723121 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711523    4864 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:41:27.723121 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711526    4864 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:41:27.723121 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711528    4864 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:41:27.723121 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711531    4864 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:41:27.723121 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711533    4864 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:41:27.723121 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711536    4864 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:41:27.723121 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711539    4864 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:41:27.723121 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711542    4864 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:41:27.723121 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711544    4864 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:41:27.723121 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711547    4864 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:41:27.723121 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711549    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:41:27.723121 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711552    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:41:27.723121 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711554    4864 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:41:27.723634 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711557    4864 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:41:27.723634 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711559    4864 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:41:27.723634 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711562    4864 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:41:27.723634 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.711589    4864 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:41:27.723634 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711718    4864 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:41:27.723634 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711723    4864 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:41:27.723634 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711726    4864 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:41:27.723634 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711729    4864 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:41:27.723634 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711732    4864 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:41:27.723634 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711736    4864 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:41:27.723634 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711739    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:41:27.723634 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711742    4864 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:41:27.723634 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711745    4864 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:41:27.723634 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711747    4864 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:41:27.723634 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711750    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:41:27.723634 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711753    4864 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:41:27.724052 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711755    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:41:27.724052 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711758    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:41:27.724052 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711761    4864 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:41:27.724052 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711763    4864 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:41:27.724052 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711766    4864 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:41:27.724052 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711769    4864 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:41:27.724052 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711771    4864 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:41:27.724052 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711774    4864 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:41:27.724052 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711776    4864 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:41:27.724052 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711779    4864 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:41:27.724052 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711781    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:41:27.724052 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711784    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:41:27.724052 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711786    4864 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:41:27.724052 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711789    4864 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:41:27.724052 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711791    4864 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:41:27.724052 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711794    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:41:27.724052 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711796    4864 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:41:27.724052 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711799    4864 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:41:27.724052 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711801    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:41:27.724531 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711804    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:41:27.724531 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711806    4864 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:41:27.724531 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711809    4864 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:41:27.724531 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711811    4864 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:41:27.724531 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711814    4864 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:41:27.724531 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711817    4864 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:41:27.724531 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711819    4864 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:41:27.724531 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711823    4864 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:41:27.724531 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711825    4864 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:41:27.724531 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711828    4864 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:41:27.724531 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711830    4864 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:41:27.724531 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711833    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:41:27.724531 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711835    4864 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:41:27.724531 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711838    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:41:27.724531 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711840    4864 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:41:27.724531 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711843    4864 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:41:27.724531 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711845    4864 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:41:27.724531 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711848    4864 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:41:27.724531 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711851    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:41:27.724531 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711853    4864 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:41:27.725020 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711856    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:41:27.725020 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711859    4864 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:41:27.725020 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711861    4864 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:41:27.725020 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711864    4864 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:41:27.725020 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711866    4864 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:41:27.725020 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711868    4864 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:41:27.725020 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711871    4864 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:41:27.725020 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711874    4864 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:41:27.725020 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711876    4864 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:41:27.725020 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711879    4864 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:41:27.725020 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711881    4864 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:41:27.725020 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711884    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:41:27.725020 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711886    4864 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:41:27.725020 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711889    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:41:27.725020 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711891    4864 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:41:27.725020 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711894    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:41:27.725020 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711896    4864 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:41:27.725020 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711899    4864 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:41:27.725020 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711903    4864 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:41:27.725614 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711906    4864 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:41:27.725614 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711910    4864 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:41:27.725614 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711913    4864 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:41:27.725614 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711916    4864 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:41:27.725614 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711918    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:41:27.725614 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711921    4864 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:41:27.725614 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711923    4864 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:41:27.725614 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711927    4864 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:41:27.725614 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711930    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:41:27.725614 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711935    4864 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:41:27.725614 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711939    4864 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:41:27.725614 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711941    4864 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:41:27.725614 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711944    4864 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:41:27.725614 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711947    4864 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:41:27.725614 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711949    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:41:27.725614 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711952    4864 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:41:27.725614 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711955    4864 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:41:27.725614 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711957    4864 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:41:27.725614 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711960    4864 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:41:27.726132 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711963    4864 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:41:27.726132 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711965    4864 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:41:27.726132 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711968    4864 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:41:27.726132 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711970    4864 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:41:27.726132 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711973    4864 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:41:27.726132 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711975    4864 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:41:27.726132 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711978    4864 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:41:27.726132 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711981    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:41:27.726132 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711983    4864 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:41:27.726132 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711986    4864 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:41:27.726132 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711988    4864 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:41:27.726132 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711991    4864 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:41:27.726132 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711993    4864 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:41:27.726132 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711996    4864 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:41:27.726132 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.711999    4864 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:41:27.726132 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712003    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:41:27.726132 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712005    4864 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:41:27.726552 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.712028    4864 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:41:27.726552 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712058    4864 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:41:27.726552 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712062    4864 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:41:27.726552 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712065    4864 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:41:27.726552 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712080    4864 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:41:27.726552 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712083    4864 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:41:27.726552 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712089    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:41:27.726552 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712092    4864 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:41:27.726552 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712095    4864 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:41:27.726552 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712098    4864 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:41:27.726552 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712101    4864 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:41:27.726552 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712103    4864 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:41:27.726552 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712106    4864 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:41:27.726552 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712109    4864 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:41:27.726552 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712112    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:41:27.726552 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712114    4864 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:41:27.726982 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712117    4864 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:41:27.726982 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712119    4864 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:41:27.726982 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712122    4864 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:41:27.726982 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712125    4864 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:41:27.726982 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712127    4864 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:41:27.726982 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712130    4864 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:41:27.726982 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712132    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:41:27.726982 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712135    4864 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:41:27.726982 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712140    4864 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:41:27.726982 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712143    4864 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:41:27.726982 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712145    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:41:27.726982 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712148    4864 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:41:27.726982 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712150    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:41:27.726982 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712153    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:41:27.726982 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712156    4864 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:41:27.726982 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712159    4864 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:41:27.726982 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712162    4864 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:41:27.726982 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712164    4864 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:41:27.726982 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712167    4864 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:41:27.726982 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712169    4864 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:41:27.727507 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712172    4864 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:41:27.727507 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712175    4864 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:41:27.727507 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712179    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:41:27.727507 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712182    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:41:27.727507 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712184    4864 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:41:27.727507 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712187    4864 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:41:27.727507 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712190    4864 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:41:27.727507 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712192    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:41:27.727507 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712195    4864 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:41:27.727507 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712197    4864 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:41:27.727507 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712200    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:41:27.727507 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712203    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:41:27.727507 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712205    4864 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:41:27.727507 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712208    4864 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:41:27.727507 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712210    4864 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:41:27.727507 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712213    4864 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:41:27.727507 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712216    4864 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:41:27.727507 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712219    4864 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:41:27.727507 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712221    4864 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:41:27.727507 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712224    4864 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:41:27.728212 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712228    4864 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:41:27.728212 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712231    4864 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:41:27.728212 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712234    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:41:27.728212 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712236    4864 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:41:27.728212 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712239    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:41:27.728212 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712241    4864 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:41:27.728212 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712244    4864 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:41:27.728212 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712247    4864 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:41:27.728212 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712250    4864 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:41:27.728212 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712253    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:41:27.728212 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712256    4864 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:41:27.728212 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712258    4864 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:41:27.728212 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712261    4864 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:41:27.728212 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712264    4864 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:41:27.728212 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712267    4864 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:41:27.728212 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712270    4864 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:41:27.728212 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712272    4864 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:41:27.728212 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712275    4864 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:41:27.728212 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712278    4864 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:41:27.728212 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712281    4864 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:41:27.728705 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712283    4864 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:41:27.728705 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712286    4864 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:41:27.728705 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712288    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:41:27.728705 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712291    4864 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:41:27.728705 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712293    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:41:27.728705 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712296    4864 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:41:27.728705 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712299    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:41:27.728705 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712301    4864 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:41:27.728705 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712304    4864 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:41:27.728705 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712309    4864 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:41:27.728705 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712313    4864 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:41:27.728705 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712316    4864 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:41:27.728705 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712321    4864 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:41:27.728705 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712324    4864 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:41:27.728705 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712327    4864 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:41:27.728705 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712330    4864 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:41:27.728705 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712333    4864 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:41:27.728705 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712336    4864 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:41:27.728705 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712340    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:41:27.729448 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712343    4864 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:41:27.729448 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712346    4864 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:41:27.729448 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712349    4864 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:41:27.729448 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712352    4864 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:41:27.729448 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712354    4864 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:41:27.729448 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712358    4864 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:41:27.729448 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712361    4864 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:41:27.729448 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712365    4864 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:41:27.729448 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712367    4864 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:41:27.729448 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712370    4864 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:41:27.729448 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712373    4864 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:41:27.729448 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:27.712376    4864 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:41:27.729448 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.712431    4864 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 00:41:27.729448 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.713430    4864 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 00:41:27.729448 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.713482    4864 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 00:41:27.729448 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.713659    4864 server.go:1014] "Starting client certificate rotation"
Oct 30 00:41:27.729448 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.713802    4864 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 00:41:27.730173 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.713845    4864 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:35:48 +0000 UTC" deadline="2028-06-06 03:55:25.880866825 +0000 UTC"
Oct 30 00:41:27.730173 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.713873    4864 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14043h13m58.166995782s"
Oct 30 00:41:27.730173 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.714109    4864 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:41:27.730173 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.719176    4864 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:41:27.730173 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.720517    4864 log.go:25] "Validated CRI v1 runtime API"
Oct 30 00:41:27.730173 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.722358    4864 log.go:25] "Validated CRI v1 image API"
Oct 30 00:41:27.730173 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.723151    4864 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 00:41:27.730173 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.725011    4864 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ef999e16-7a0f-4eab-9b2c-4e9ab340269a:/dev/nvme0n1p3 f6480457-aeeb-4a23-830e-3c364a92c5fe:/dev/nvme0n1p4]
Oct 30 00:41:27.730514 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.725030    4864 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d56c1057633e403e87c7a5c42194323a27dbf826d9452b3a35c0ed5b3382be6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d56c1057633e403e87c7a5c42194323a27dbf826d9452b3a35c0ed5b3382be6/userdata/shm major:0 minor:307 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1940788f530ef326fd10fac6b6036bada7821b1156f1720b6ab940debdea870d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1940788f530ef326fd10fac6b6036bada7821b1156f1720b6ab940debdea870d/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1aa45a060b9b9d4d37b976fdbc291e098d6c2fb0cde1b742e32f4c411b49b01c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1aa45a060b9b9d4d37b976fdbc291e098d6c2fb0cde1b742e32f4c411b49b01c/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2b3fba6489f614f481bcba72e9e08c34e049a7410f507c517a286b12792bcf13/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2b3fba6489f614f481bcba72e9e08c34e049a7410f507c517a286b12792bcf13/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d93152dd6b759165b13a9074c7ea4ba5e69985e2eb6f607478de2cb748f749d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d93152dd6b759165b13a9074c7ea4ba5e69985e2eb6f607478de2cb748f749d/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3343d6e6f697450771e01c175f1ff91f886fdd06afd1f8079dfff032504d5282/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3343d6e6f697450771e01c175f1ff91f886fdd06afd1f8079dfff032504d5282/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3345ad6714170a3d6ebec355a44fed3839dc923d622aba74687544a5c9cfe9f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3345ad6714170a3d6ebec355a44fed3839dc923d622aba74687544a5c9cfe9f2/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/361e853b438a0235d45cedd91283a0588a709a51d75f46c2bf559651514d776b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/361e853b438a0235d45cedd91283a0588a709a51d75f46c2bf559651514d776b/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3cc619806395200599c333ff4d975468fa2183cea96e8de9ac4cdbc083843bb3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3cc619806395200599c333ff4d975468fa2183cea96e8de9ac4cdbc083843bb3/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/488f9269c3c668a4e1d441fe21955db3244c963444ad131d77de0c1a82ae847d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/488f9269c3c668a4e1d441fe21955db3244c963444ad131d77de0c1a82ae847d/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a387f3dc74cf8bf7a52fa606628b8212707b6a2bc64826138ea18bb1e2b93b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a387f3dc74cf8bf7a52fa606628b8212707b6a2bc64826138ea18bb1e2b93b6/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53018d64b73fdb4a068c8ca329984997d649894a8bd29be62d2a0095309671b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53018d64b73fdb4a068c8ca329984997d649894a8bd29be62d2a0095309671b4/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5dcb777893bb0ed143391b3d0b0eb4304f96ecbef22135c33ea430db6e964cf0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5dcb777893bb0ed143391b3d0b0eb4304f96ecbef22135c33ea430db6e964cf0/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ebac1c448b6d1cd408bc250e18f054eb6714965646f9d0411ca4c508f4ec837/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ebac1c448b6d1cd408bc250e18f054eb6714965646f9d0411ca4c508f4ec837/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/60513a52513278b2ffcb4c04eeda28a50d46522ccefecac86d43bf4faa41f6ee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/60513a52513278b2ffcb4c04eeda28a50d46522ccefecac86d43bf4faa41f6ee/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/666786686b632a96170c0208d19941a722053a47b1c493774fcf7227ea91797c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/666786686b632a96170c0208d19941a722053a47b1c493774fcf7227ea91797c/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ff453ef83e8b4374463dbceb3eda86c51c7b10b2b23e81a7aabffa9367d65c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ff453ef83e8b4374463dbceb3eda86c51c7b10b2b23e81a7aabffa9367d65c2/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4be861396077b7336d341d30d02aa1c3b1d68b59eabd0533b5a32bf58a784fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4be861396077b7336d341d30d02aa1c3b1d68b59eabd0533b5a32bf58a784fc/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e00c480be6b0794693cf776efc17b9266d72737419c9ab17343e8f19abe31439/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e00c480be6b0794693cf776efc17b9266d72737419c9ab17343e8f19abe31439/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4346845caaf9b198310d004c52c7d155f38f4caf38a4b1079ff4739a59b3406/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4346845caaf9b198310d004c52c7d155f38f4caf38a4b1079ff4739a59b3406/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f14f1ab09e47b335bc8ab3c39be0ac2a6416c96143fec1488db1a8266128472d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f14f1ab09e47b335bc8ab3c39be0ac2a6416c96143fec1488db1a8266128472d/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff40dff7498f99566d9b280ec92caa89e34a8eba8e4a13a0258b6df543ba8747/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff40dff7498f99566d9b280ec92caa89e34a8eba8e4a13a0258b6df543ba8747/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0142f4ce-76c5-4ca1-ac00-c2c334696905/volumes/kubernetes.io~projected/kube-api-access-45zv7:{mountpoint:/var/lib/kubelet/pods/0142f4ce-76c5-4ca1-ac00-c2c334696905/volumes/kubernetes.io~projected/kube-api-access-45zv7 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0142f4ce-76c5-4ca1-ac00-c2c334696905/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/0142f4ce-76c5-4ca1-ac00-c2c334696905/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a50fdaf-7855-4033-96c8-6cb23212a986/volumes/kubernetes.io~projected/kube-api-access-pxvpz:{mountpoint:/var/lib/kubelet/pods/0a50fdaf-7855-4033-96c8-6cb23212a986/volumes/kubernetes.io~projected/kube-api-access-pxvpz major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493/volumes/kubernetes.io~projected/kube-api-access-kksvb:{mountpoint:/var/lib/kubelet/pods/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493/volumes/kubernetes.io~projected/kube-api-access-kksvb major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0eac4baa-d0d9-43c0-8f81-3733a026e091/volumes/kubernetes.io~projected/kube-api-access-nnf9q:{mountpoint:/var/lib/kubelet/pods/0eac4baa-d0d9-43c0-8f81-3733a026e091/volumes/kubernetes.io~projected/kube-api-access-nnf9q major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19a807d5-c007-47b4-953c-28de606ababc/volumes/kubernetes.io~projected/kube-api-access-8nkxz:{mountpoint:/var/lib/kubelet/pods/19a807d5-c007-47b4-953c-28de606ababc/volumes/kubernetes.io~projected/kube-api-access-8nkxz major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/204212cc-1c4c-4bfb-8b41-03a98042166c/volumes/kubernetes.io~projected/kube-api-access-7vvj2:{mountpoint:/var/lib/kubelet/pods/204212cc-1c4c-4bfb-8b41-03a98042166c/volumes/kubernetes.io~projected/kube-api-access-7vvj2 major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ac86565-2bfb-45e4-840d-c0f3a29cdfa5/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/2ac86565-2bfb-45e4-840d-c0f3a29cdfa5/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc/volumes/kubernetes.io~projected/kube-api-access-v7glf:{mountpoint:/var/lib/kubelet/pods/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc/volumes/kubernetes.io~projected/kube-api-access-v7glf major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/380df728-1bf4-4258-ba04-c31e1d28a3aa/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/380df728-1bf4-4258-ba04-c31e1d28a3aa/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/380df728-1bf4-4258-ba04-c31e1d28a3aa/volumes/kubernetes.io~projected/kube-api-access-j89wv:{mountpoint:/var/lib/kubelet/pods/380df728-1bf4-4258-ba04-c31e1d28a3aa/volumes/kubernetes.io~projected/kube-api-access-j89wv major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/380df728-1bf4-4258-ba04-c31e1d28a3aa/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/380df728-1bf4-4258-ba04-c31e1d28a3aa/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/433f04c0-a9ea-45c8-a2b7-245980607391/volumes/kubernetes.io~projected/kube-api-access-4q94d:{mountpoint:/var/lib/kubelet/pods/433f04c0-a9ea-45c8-a2b7-245980607391/volumes/kubernetes.io~projected/kube-api-access-4q94d major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/433f04c0-a9ea-45c8-a2b7-245980607391/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/433f04c0-a9ea-45c8-a2b7-245980607391/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4509811e-30a0-4c8b-b21e-58826bd1f6c6/volumes/kubernetes.io~projected/kube-api-access-5tkrk:{mountpoint:/var/lib/kubelet/pods/4509811e-30a0-4c8b-b21e-58826bd1f6c6/volumes/kubernetes.io~projected/kube-api-access-5tkrk major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4770276e-e9dc-4f6c-94f6-d2dfe2f6f115/volumes/kubernetes.io~projected/kube-api-access-lwnlf:{mountpoint:/var/lib/kubelet/pods/4770276e-e9dc-4f6c-94f6-d2dfe2f6f115/volumes/kubernetes.io~projected/kube-api-access-lwnlf major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/564f9ba3-d19e-4cac-9021-81939eb8e631/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/564f9ba3-d19e-4cac-9021-81939eb8e631/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/564f9ba3-d19e-4cac-9021-81939eb8e631/volumes/kubernetes.io~projected/kube-api-access-d8s6t:{mountpoint:/var/lib/kubelet/pods/564f9ba3-d19e-4cac-9021-81939eb8e631/volumes/kubernetes.io~projected/kube-api-access-d8s6t major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/564f9ba3-d19e-4cac-9021-81939eb8e631/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/564f9ba3-d19e-4cac-9021-81939eb8e631/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/564f9ba3-d19e-4cac-9021-81939eb8e631/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/564f9ba3-d19e-4cac-9021-81939eb8e631/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59e5f12b-4a3e-449b-b285-17747780e1cf/volumes/kubernetes.io~projected/kube-api-access-cmm4r:{mountpoint:/var/lib/kubelet/pods/59e5f12b-4a3e-449b-b285-17747780e1cf/volumes/kubernetes.io~projected/kube-api-access-cmm4r major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59e5f12b-4a3e-449b-b285-17747780e1cf/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/59e5f12b-4a3e-449b-b285-17747780e1cf/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76b2bcb8-636c-46b5-8dfb-95831470349e/volumes/kubernetes.io~projected/kube-api-access-r64gt:{mountpoint:/var/lib/kubelet/pods/76b2bcb8-636c-46b5-8dfb-95831470349e/volumes/kubernetes.io~projected/kube-api-access-r64gt major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/794aa402-53bf-43fc-b21b-a176bbbcd132/volumes/kubernetes.io~projected/kube-api-access-79mmb:{mountpoint:/var/lib/kubelet/pods/794aa402-53bf-43fc-b21b-a176bbbcd132/volumes/kubernetes.io~projected/kube-api-access-79mmb major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e4dbba6-28af-46b3-ac90-37ce437fd318/volumes/kubernetes.io~projected/kube-api-access-sljzz:{mountpoint:/var/lib/kubelet/pods/8e4dbba6-28af-46b3-ac90-37ce437fd318/volumes/kubernetes.io~projected/kube-api-access-sljzz major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e4dbba6-28af-46b3-ac90-37ce437fd318/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8e4dbba6-28af-46b3-ac90-37ce437fd318/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55/volumes/kubernetes.io~projected/kube-api-access-54t8p:{mountpoint:/var/lib/kubelet/pods/a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55/volumes/kubernetes.io~projected/kube-api-access-54t8p major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55/volumes/kubernetes.io~secret/cert major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2dadf26-b255-4281-91f4-98536d370d21/volumes/kubernetes.io~projected/kube-api-access-s72cg:{mountpoint:/var/lib/kubelet/pods/a2dadf26-b255-4281-91f4-98536d370d21/volumes/kubernetes.io~projected/kube-api-access-s72cg major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae0f3040-049e-4f4e-a6e5-206b31aeea55/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/ae0f3040-049e-4f4e-a6e5-206b31aeea55/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae0f3040-049e-4f4e-a6e5-206b31aeea55/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/ae0f3040-049e-4f4e-a6e5-206b31aeea55/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae0f3040-049e-4f4e-a6e5-206b31aeea55/volumes/kubernetes.io~projected/kube-api-access-jpqz6:{mountpoint:/var/lib/kubelet/pods/ae0f3040-049e-4f4e-a6e5-206b31aeea55/volumes/kubernetes.io~projected/kube-api-access-jpqz6 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbef968a-7aa4-4e23-9278-3706e60d8c0b/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/bbef968a-7aa4-4e23-9278-3706e60d8c0b/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c48ee8c6-60b2-4229-a013-4199c44813e4/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/c48ee8c6-60b2-4229-a013-4199c44813e4/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c48ee8c6-60b2-4229-a013-4199c44813e4/volumes/kubernetes.io~projected/kube-api-access-g77n6:{mountpoint:/var/lib/kubelet/pods/c48ee8c6-60b2-4229-a013-4199c44813e4/volumes/kubernetes.io~projected/kube-api-access-g77n6 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c48ee8c6-60b2-4229-a013-4199c44813e4/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/c48ee8c6-60b2-4229-a013-4199c44813e4/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c48ee8c6-60b2-4229-a013-4199c44813e4/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/c48ee8c6-60b2-4229-a013-4199c44813e4/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c75bc9c9-74be-412b-a5ac-794f971bea31/volumes/kubernetes.io~projected/kube-api-access-jq6ls:{mountpoint:/var/lib/kubelet/pods/c75bc9c9-74be-412b-a5ac-794f971bea31/volumes/kubernetes.io~projected/kube-api-access-jq6ls major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc/volumes/kubernetes.io~projected/kube-api-access-75l4z:{mountpoint:/var/lib/kubelet/pods/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc/volumes/kubernetes.io~projected/kube-api-access-75l4z major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d95535e9-e677-463a-a662-7e8045c71d9b/volumes/kubernetes.io~projected/kube-api-access-78557:{mountpoint:/var/lib/kubelet/pods/d95535e9-e677-463a-a662-7e8045c71d9b/volumes/kubernetes.io~projected/kube-api-access-78557 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d95535e9-e677-463a-a662-7e8045c71d9b/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/d95535e9-e677-463a-a662-7e8045c71d9b/volumes/kubernetes.io~secret/default-certificate major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d95535e9-e677-463a-a662-7e8045c71d9b/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/d95535e9-e677-463a-a662-7e8045c71d9b/volumes/kubernetes.io~secret/stats-auth major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e05b9b53-5f5b-402e-8599-ce61481346e6/volumes/kubernetes.io~projected/kube-api-access-l8xgb:{mountpoint:/var/lib/kubelet/pods/e05b9b53-5f5b-402e-8599-ce61481346e6/volumes/kubernetes.io~projected/kube-api-access-l8xgb major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074/volumes/kubernetes.io~projected/kube-api-access-bpn92:{mountpoint:/var/lib/kubelet/pods/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074/volumes/kubernetes.io~projected/kube-api-access-bpn92 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5fade25-18f0-4e3f-978a-5bb39a356ab0/volumes/kubernetes.io~projected/kube-api-access-tk7mm:{mountpoint:/var/lib/kubelet/pods/f5fade25-18f0-4e3f-978a-5bb39a356ab0/volumes/kubernetes.io~projected/kube-api-access-tk7mm major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc3514a7-8b72-40ff-b45a-1c2005b59360/volumes/kubernetes.io~projected/kube-api-access-cpx52:{mountpoint:/var/lib/kubelet/pods/fc3514a7-8b72-40ff-b45a-1c2005b59360/volumes/kubernetes.io~projected/kube-api-access-cpx52 major:0 minor:210 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/0bc411f119831f16bc155b558d5ac5acaf563ba1d8151ffbaa6b139d99009d5d/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/59a9b326b6c8ce241fcc6169e67159543b10a14e9219727e771982cd582568a1/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/2c454cd7ca49b4b733c398cc4be1f7d7935918313cd24fd033e387a802ce16f5/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/ad16d5d64c1fb607c07fd9b50f24bfae86268317ac5d75fbb6c2e0181e9e9bdc/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/f417431526f115ccae31c682b9d19e491c369a1a6275c5b41a949f03e953991f/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/06d24b272e7e71b3c9b6b34bfd38e64cb23e52d249c4d46af8285beee514aa29/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/161f72f378fcc741cde255f47f7c59d93a1b3f9183b71ba724c89930dc5894b4/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/f93c4d237d37e7a51133ce1e3a30e5cac33d99e76dfbf311b0ff10ac13a01531/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/749eb5f18be1b28fbc38344a22acb9564c1bc59db2bad6287487b3223ab248f6/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/11152b1c552f52bfdf23471332f88170e9c83524cf4417b16d6d480e8f13bb7e/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-135:{mountpoint:/var/lib/containers/storage/overlay/a2ea10e17f1162b09bce744c9bd715fd2734c7f6c5ea12d425fe4f3ecc2dee5d/merged major:0 minor:135 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/139f298089b2066e2663b4a49860ed514bcab3d70ee4467836f7b130ee3c06ff/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/4d21507af9aa94fa164a86d9e97b7cf4fdaa571932eb1db05b167eae2637bf2e/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/5de5e4d8ba525731e26a70514f3b4617ab5ef4e5c499106fcbc85c8e5974b41a/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/0e3091d6e205b6a1edd3757402b8e74d4ec12d224235356e299d5bf020d05f84/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/469c126950a690110fbc80b22f5ac87c6bab43adeb5d0d676d3db594d72227c3/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/a9029bdf305dcd788c51eb205dad297b6d074b6425ef800213a0a1b511278457/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/65fbb34a44a8b205b7e13c03e782a46f9fe14b96b137690d2c6a1c2b60909273/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/075f2c438cb55bed93da03740efb85c14d86d490004f30ad8db80a4b3a21f1b8/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/cab1f5af82ba282678ee14c134255db5fc7d0a7b095db91eb412891f30ed395c/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/057abbf34d4732d2a70014a3c5001a76690f9aa983a8fe30560fd10387918432/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/45bea7e039a3dd54dd21fb36fa1d1bfc1465f319359c93918d2c67bb6d76e857/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/41d0625864b8887b0aa9e6adf1dfcf854d04776499b48a2c627bbf24814e7267/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/6082bdaa5199425151a1a12b6c81ad81416a6403feec0d69b41642804e908378/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/503118135eb8a6ef65a2bf628fbd4abe919f822e680e95caabbfd5bf94282d52/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/fb155904e5cc289920ed5921d7c77ae297557d104e5964cab3f974bcbfd4cf3c/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/426f2a641e30fd2e4173be5f77b0f9db3a89afa147f22ee9469c90d1900d834d/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/7eda818a4efa8897c10d9e17dc2a70bd027df3e170538ade6599ceb8c0ba4e4a/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/f7e381a7abd81d46a9ac44ced3cf21e640c12a4277b23da5b060bd54c7717117/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/161a6f108f8eae54d950b15f4ad7dc4dbf2ea97db3bf91b06856a753e791e7ca/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/1676fd984f412cfe89e614c1b18949612e7ae165b3e9622aea69aeec34a67116/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/7d6c54db299c51ad624f9e98fd3742f146ee5f3981be44ba7949c3542b42fd4a/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/c09bbe154af8af6efdc0a74d1ebb9d895220c50ec5d8b494ffddbc0b47bbcdb4/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/1afd56bd20c6d031ae8cd461d31ea6e38ce3da18c718f0c5ed7cc4b6214f01f0/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/15515d589e9ef5d53ab8977a0a7386c36eed4a536a3903a5462bf92a972a7691/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/889830764bf3458851c8044b1c93d8e2680b4ba9cff755be16d5019010c06175/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/7bdde61b9f65b8d218d705b0870140ed1d42b14e6f0d17eee41a8fb30374df91/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/71fc505236fcfe3573c242c0ff3163fa2440104db7020c4fc7914136cb22f4b3/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/07341497ad10aa0a4e618fce79843a60a0946fd06daf9be73aeeb1bd5476bfe5/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/a843020d4d199ddceb6f0098e8f0e70a76c10350a9c2776c55775d591295fa3a/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/34f6740e32cd9eaf8e609b9666d650805ab1b8fca53e5ec87bb54ed28fc8c86e/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/a6a96569a6a10adb781d19749a2a64238db83aa9494b777547c50aa635db23d3/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/e7eb5633025d178331e7f51752bbeb9a34dd171f08dd6b39960283916adfb0ea/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/6c20942dced0d4db1f084d8b789e579b374759f1440634bbfd875b328df41d86/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/3a1cdd79a325a51756afe538e503c391e751df5697b81cc3239544aaa5324211/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/0d16691be7ff6d9d4ca008dd284feb0ce0aab527a65b1f01b248a89c240a848a/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/f94f2c806496d03a342c4df91acc38f94f6875da8e383210afb804d481fa6344/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/fca83dbfcf1c6e531489bef7e449ae9aad378862086fb5325ea6712397aba5fc/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/b7383ca60dff5ee75570d26faf5c61c1c33b124956b74875aaf86c50784e0210/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/4a34ab743cfae2e38d67eaafc1e4df53116f9ac3bb6e3c7253ee80bd792fba78/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/d888fe450dfc4ad11c89d2be653c6a6ebcc309e32e2a7ce68828ccba8e3fafa3/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/a97ec52ceb0fc4c2af8117c9a99e8265893bc6a9ed1e9a284a29090a7a1ed0c2/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/0a269de1826692bc8fc0efb2647c86f23dd3a2d2d2af70abf3e3a294a41c0864/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/4a243845bee6bb5087a9d7eab46ef3a8d00315f8a37db1818b439980af1b99bf/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 00:41:27.734418 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.734039    4864 manager.go:217] Machine: {Timestamp:2026-10-30 00:41:27.733566119 +0000 UTC m=+0.071668024 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3078802 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec200a57591a5ecf36a35b4f290abc7e SystemUUID:ec200a57-591a-5ecf-36a3-5b4f290abc7e BootID:0744abf2-5a97-4d46-b8a1-995f2b472c34 Filesystems:[{Device:/var/lib/kubelet/pods/19a807d5-c007-47b4-953c-28de606ababc/volumes/kubernetes.io~projected/kube-api-access-8nkxz DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff40dff7498f99566d9b280ec92caa89e34a8eba8e4a13a0258b6df543ba8747/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d93152dd6b759165b13a9074c7ea4ba5e69985e2eb6f607478de2cb748f749d/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1aa45a060b9b9d4d37b976fdbc291e098d6c2fb0cde1b742e32f4c411b49b01c/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/59e5f12b-4a3e-449b-b285-17747780e1cf/volumes/kubernetes.io~projected/kube-api-access-cmm4r DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53018d64b73fdb4a068c8ca329984997d649894a8bd29be62d2a0095309671b4/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3345ad6714170a3d6ebec355a44fed3839dc923d622aba74687544a5c9cfe9f2/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3cc619806395200599c333ff4d975468fa2183cea96e8de9ac4cdbc083843bb3/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/380df728-1bf4-4258-ba04-c31e1d28a3aa/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/361e853b438a0235d45cedd91283a0588a709a51d75f46c2bf559651514d776b/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3343d6e6f697450771e01c175f1ff91f886fdd06afd1f8079dfff032504d5282/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/433f04c0-a9ea-45c8-a2b7-245980607391/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a387f3dc74cf8bf7a52fa606628b8212707b6a2bc64826138ea18bb1e2b93b6/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76b2bcb8-636c-46b5-8dfb-95831470349e/volumes/kubernetes.io~projected/kube-api-access-r64gt DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ac86565-2bfb-45e4-840d-c0f3a29cdfa5/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493/volumes/kubernetes.io~projected/kube-api-access-kksvb DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ff453ef83e8b4374463dbceb3eda86c51c7b10b2b23e81a7aabffa9367d65c2/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0142f4ce-76c5-4ca1-ac00-c2c334696905/volumes/kubernetes.io~projected/kube-api-access-45zv7 DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c75bc9c9-74be-412b-a5ac-794f971bea31/volumes/kubernetes.io~projected/kube-api-access-jq6ls DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/564f9ba3-d19e-4cac-9021-81939eb8e631/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/204212cc-1c4c-4bfb-8b41-03a98042166c/volumes/kubernetes.io~projected/kube-api-access-7vvj2 DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1940788f530ef326fd10fac6b6036bada7821b1156f1720b6ab940debdea870d/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/564f9ba3-d19e-4cac-9021-81939eb8e631/volumes/kubernetes.io~projected/kube-api-access-d8s6t DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/433f04c0-a9ea-45c8-a2b7-245980607391/volumes/kubernetes.io~projected/kube-api-access-4q94d DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/666786686b632a96170c0208d19941a722053a47b1c493774fcf7227ea91797c/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0eac4baa-d0d9-43c0-8f81-3733a026e091/volumes/kubernetes.io~projected/kube-api-access-nnf9q DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d95535e9-e677-463a-a662-7e8045c71d9b/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f5fade25-18f0-4e3f-978a-5bb39a356ab0/volumes/kubernetes.io~projected/kube-api-access-tk7mm DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/564f9ba3-d19e-4cac-9021-81939eb8e631/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/380df728-1bf4-4258-ba04-c31e1d28a3aa/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0a50fdaf-7855-4033-96c8-6cb23212a986/volumes/kubernetes.io~projected/kube-api-access-pxvpz DeviceMajor:0 DeviceMinor:334 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f14f1ab09e47b335bc8ab3c39be0ac2a6416c96143fec1488db1a8266128472d/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/564f9ba3-d19e-4cac-9021-81939eb8e631/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/60513a52513278b2ffcb4c04eeda28a50d46522ccefecac86d43bf4faa41f6ee/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074/volumes/kubernetes.io~projected/kube-api-access-bpn92 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ae0f3040-049e-4f4e-a6e5-206b31aeea55/volumes/kubernetes.io~projected/kube-api-access-jpqz6 DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/380df728-1bf4-4258-ba04-c31e1d28a3aa/volumes/kubernetes.io~projected/kube-api-access-j89wv DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2b3fba6489f614f481bcba72e9e08c34e049a7410f507c517a286b12792bcf13/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4509811e-30a0-4c8b-b21e-58826bd1f6c6/volumes/kubernetes.io~projected/kube-api-access-5tkrk DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ae0f3040-049e-4f4e-a6e5-206b31aeea55/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ae0f3040-049e-4f4e-a6e5-206b31aeea55/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c48ee8c6-60b2-4229-a013-4199c44813e4/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a2dadf26-b255-4281-91f4-98536d370d21/volumes/kubernetes.io~projected/kube-api-access-s72cg DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e05b9b53-5f5b-402e-8599-ce61481346e6/volumes/kubernetes.io~projected/kube-api-access-l8xgb DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc/volumes/kubernetes.io~projected/kube-api-access-75l4z DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55/volumes/kubernetes.io~projected/kube-api-access-54t8p DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/4770276e-e9dc-4f6c-94f6-d2dfe2f6f115/volumes/kubernetes.io~projected/kube-api-access-lwnlf DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e4dbba6-28af-46b3-ac90-37ce437fd318/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c48ee8c6-60b2-4229-a013-4199c44813e4/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5dcb777893bb0ed143391b3d0b0eb4304f96ecbef22135c33ea430db6e964cf0/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-135 DeviceMajor:0 DeviceMinor:135 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc3514a7-8b72-40ff-b45a-1c2005b59360/volumes/kubernetes.io~projected/kube-api-access-cpx52 DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/488f9269c3c668a4e1d441fe21955db3244c963444ad131d77de0c1a82ae847d/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d56c1057633e403e87c7a5c42194323a27dbf826d9452b3a35c0ed5b3382be6/userdata/shm DeviceMajor:0 DeviceMinor:307 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc/volumes/kubernetes.io~projected/kube-api-access-v7glf DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8e4dbba6-28af-46b3-ac90-37ce437fd318/volumes/kubernetes.io~projected/kube-api-access-sljzz DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/794aa402-53bf-43fc-b21b-a176bbbcd132/volumes/kubernetes.io~projected/kube-api-access-79mmb DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d95535e9-e677-463a-a662-7e8045c71d9b/volumes/kubernetes.io~projected/kube-api-access-78557 DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4be861396077b7336d341d30d02aa1c3b1d68b59eabd0533b5a32bf58a784fc/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4346845caaf9b198310d004c52c7d155f38f4caf38a4b1079ff4739a59b3406/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bbef968a-7aa4-4e23-9278-3706e60d8c0b/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e00c480be6b0794693cf776efc17b9266d72737419c9ab17343e8f19abe31439/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d95535e9-e677-463a-a662-7e8045c71d9b/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ebac1c448b6d1cd408bc250e18f054eb6714965646f9d0411ca4c508f4ec837/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/59e5f12b-4a3e-449b-b285-17747780e1cf/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c48ee8c6-60b2-4229-a013-4199c44813e4/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0142f4ce-76c5-4ca1-ac00-c2c334696905/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c48ee8c6-60b2-4229-a013-4199c44813e4/volumes/kubernetes.io~projected/kube-api-access-g77n6 DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0d56c1057633e40 MacAddress:92:80:62:f9:b8:fa Speed:10000 Mtu:8901} {Name:2d93152dd6b7591 MacAddress:0e:52:31:86:85:c2 Speed:10000 Mtu:8901} {Name:3343d6e6f697450 MacAddress:4a:e0:34:d6:97:48 Speed:10000 Mtu:8901} {Name:3345ad6714170a3 MacAddress:12:3e:ad:17:31:f4 Speed:10000 Mtu:8901} {Name:361e853b438a023 MacAddress:96:51:f1:23:af:8b Speed:10000 Mtu:8901} {Name:3cc619806395200 MacAddress:86:86:78:fd:60:3a Speed:10000 Mtu:8901} {Name:488f9269c3c668a MacAddress:56:71:28:18:58:49 Speed:10000 Mtu:8901} {Name:53018d64b73fdb4 MacAddress:3e:5a:c1:af:55:7b Speed:10000 Mtu:8901} {Name:5dcb777893bb0ed MacAddress:ce:1d:a3:fe:e8:11 Speed:10000 Mtu:8901} {Name:60513a52513278b MacAddress:f6:8d:45:60:db:47 Speed:10000 Mtu:8901} {Name:a4be861396077b7 MacAddress:8a:14:7b:34:0b:c2 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d8:17:b3:6b Speed:0 Mtu:9001} {Name:br-int MacAddress:de:58:20:33:e1:5c Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:d8:17:b3:6b Speed:0 Mtu:9001} {Name:ff40dff7498f995 MacAddress:f6:e4:9b:78:88:93 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:a6:13:f9:75:44:cd Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:52:fa:01:74:2b:3c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 00:41:27.734665 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.734422    4864 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 00:41:27.734665 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.734468    4864 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 00:41:27.734842 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.734814    4864 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 00:41:27.734969 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.734842    4864 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-210.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 00:41:27.735014 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.734971    4864 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 00:41:27.735014 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.734979    4864 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 00:41:27.735014 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.734989    4864 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:41:27.735014 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.735002    4864 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:41:27.735145 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.735098    4864 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 00:41:27.735285 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.735275    4864 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 00:41:27.735391 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.735382    4864 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 00:41:27.735420 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.735416    4864 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 00:41:27.735450 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.735428    4864 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 00:41:27.735450 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.735438    4864 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 00:41:27.735450 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.735446    4864 kubelet.go:405] "Adding apiserver pod source"
Oct 30 00:41:27.735545 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.735454    4864 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 00:41:27.736427 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.736411    4864 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 00:41:27.736427 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.736424    4864 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:41:27.736894 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.736874    4864 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 00:41:27.739324 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.739311    4864 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 00:41:27.739324 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.739324    4864 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:41:27.739422 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.739413    4864 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 00:41:27.739599 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.739589    4864 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 00:41:27.739635 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.739624    4864 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 00:41:27.739717 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.739708    4864 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 00:41:27.739755 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.739722    4864 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 00:41:27.739755 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.739728    4864 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 00:41:27.739755 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.739735    4864 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 00:41:27.739755 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.739741    4864 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 00:41:27.739755 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.739747    4864 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 00:41:27.739755 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.739757    4864 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 00:41:27.739905 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.739763    4864 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 00:41:27.739905 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.739771    4864 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 00:41:27.739905 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.739777    4864 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 00:41:27.739905 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.739789    4864 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 00:41:27.739905 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.739798    4864 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 00:41:27.739905 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.739820    4864 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 00:41:27.739905 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.739826    4864 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 00:41:27.743096 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.743063    4864 server.go:1263] "Started kubelet"
Oct 30 00:41:27.743175 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.743150    4864 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 00:41:27.743313 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.743256    4864 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 00:41:27.743797 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.743364    4864 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 00:41:27.743797 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.743738    4864 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 00:41:27.743639 ip-10-0-134-210 systemd[1]: Started Kubernetes Kubelet.
Oct 30 00:41:27.746223 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.745576    4864 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 00:41:27.748896 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.748577    4864 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 00:41:27.748896 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.748599    4864 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 00:41:27.748896 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.748647    4864 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 00:41:27.748896 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.748657    4864 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 00:41:27.748896 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.748656    4864 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:35:50 +0000 UTC" deadline="2028-07-28 20:03:07.972568758 +0000 UTC"
Oct 30 00:41:27.748896 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.748679    4864 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15307h21m40.223892971s"
Oct 30 00:41:27.748896 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.748707    4864 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 00:41:27.748896 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:27.748749    4864 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-210.ec2.internal\" not found"
Oct 30 00:41:27.751196 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751159    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e4dbba6-28af-46b3-ac90-37ce437fd318" volumeName="kubernetes.io/secret/8e4dbba6-28af-46b3-ac90-37ce437fd318-serving-cert" seLinuxMountContext=""
Oct 30 00:41:27.751279 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751200    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a50fdaf-7855-4033-96c8-6cb23212a986" volumeName="kubernetes.io/empty-dir/0a50fdaf-7855-4033-96c8-6cb23212a986-data-volume" seLinuxMountContext=""
Oct 30 00:41:27.751279 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751214    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a50fdaf-7855-4033-96c8-6cb23212a986" volumeName="kubernetes.io/projected/0a50fdaf-7855-4033-96c8-6cb23212a986-kube-api-access-pxvpz" seLinuxMountContext=""
Oct 30 00:41:27.751279 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751227    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4770276e-e9dc-4f6c-94f6-d2dfe2f6f115" volumeName="kubernetes.io/projected/4770276e-e9dc-4f6c-94f6-d2dfe2f6f115-kube-api-access-lwnlf" seLinuxMountContext=""
Oct 30 00:41:27.751279 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751239    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="564f9ba3-d19e-4cac-9021-81939eb8e631" volumeName="kubernetes.io/configmap/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-certificates" seLinuxMountContext=""
Oct 30 00:41:27.751279 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751251    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e4dbba6-28af-46b3-ac90-37ce437fd318" volumeName="kubernetes.io/projected/8e4dbba6-28af-46b3-ac90-37ce437fd318-kube-api-access-sljzz" seLinuxMountContext=""
Oct 30 00:41:27.751279 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751263    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5fade25-18f0-4e3f-978a-5bb39a356ab0" volumeName="kubernetes.io/configmap/f5fade25-18f0-4e3f-978a-5bb39a356ab0-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 00:41:27.751279 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751275    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0142f4ce-76c5-4ca1-ac00-c2c334696905" volumeName="kubernetes.io/configmap/0142f4ce-76c5-4ca1-ac00-c2c334696905-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751289    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="204212cc-1c4c-4bfb-8b41-03a98042166c" volumeName="kubernetes.io/projected/204212cc-1c4c-4bfb-8b41-03a98042166c-kube-api-access-7vvj2" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751300    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="564f9ba3-d19e-4cac-9021-81939eb8e631" volumeName="kubernetes.io/secret/564f9ba3-d19e-4cac-9021-81939eb8e631-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751312    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59e5f12b-4a3e-449b-b285-17747780e1cf" volumeName="kubernetes.io/secret/59e5f12b-4a3e-449b-b285-17747780e1cf-serving-cert" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751323    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5fade25-18f0-4e3f-978a-5bb39a356ab0" volumeName="kubernetes.io/configmap/f5fade25-18f0-4e3f-978a-5bb39a356ab0-cni-binary-copy" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751335    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0eac4baa-d0d9-43c0-8f81-3733a026e091" volumeName="kubernetes.io/configmap/0eac4baa-d0d9-43c0-8f81-3733a026e091-cni-binary-copy" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751346    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc" volumeName="kubernetes.io/configmap/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-telemetry-config" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751373    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="564f9ba3-d19e-4cac-9021-81939eb8e631" volumeName="kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-kube-api-access-d8s6t" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751385    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c48ee8c6-60b2-4229-a013-4199c44813e4" volumeName="kubernetes.io/secret/c48ee8c6-60b2-4229-a013-4199c44813e4-installation-pull-secrets" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751392    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c75bc9c9-74be-412b-a5ac-794f971bea31" volumeName="kubernetes.io/projected/c75bc9c9-74be-412b-a5ac-794f971bea31-kube-api-access-jq6ls" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751400    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0142f4ce-76c5-4ca1-ac00-c2c334696905" volumeName="kubernetes.io/secret/0142f4ce-76c5-4ca1-ac00-c2c334696905-serving-cert" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751408    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cee8ca5c-ba97-47b5-b22f-2e161bb93fdc" volumeName="kubernetes.io/projected/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-kube-api-access-75l4z" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751415    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074" volumeName="kubernetes.io/projected/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-kube-api-access-bpn92" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751426    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="433f04c0-a9ea-45c8-a2b7-245980607391" volumeName="kubernetes.io/configmap/433f04c0-a9ea-45c8-a2b7-245980607391-config" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751438    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59e5f12b-4a3e-449b-b285-17747780e1cf" volumeName="kubernetes.io/configmap/59e5f12b-4a3e-449b-b285-17747780e1cf-config" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751449    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76b2bcb8-636c-46b5-8dfb-95831470349e" volumeName="kubernetes.io/projected/76b2bcb8-636c-46b5-8dfb-95831470349e-kube-api-access-r64gt" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751465    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e4dbba6-28af-46b3-ac90-37ce437fd318" volumeName="kubernetes.io/configmap/8e4dbba6-28af-46b3-ac90-37ce437fd318-trusted-ca" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751477    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55" volumeName="kubernetes.io/projected/a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55-kube-api-access-54t8p" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751492    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae0f3040-049e-4f4e-a6e5-206b31aeea55" volumeName="kubernetes.io/empty-dir/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-tuned" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751505    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4509811e-30a0-4c8b-b21e-58826bd1f6c6" volumeName="kubernetes.io/projected/4509811e-30a0-4c8b-b21e-58826bd1f6c6-kube-api-access-5tkrk" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751528    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0142f4ce-76c5-4ca1-ac00-c2c334696905" volumeName="kubernetes.io/projected/0142f4ce-76c5-4ca1-ac00-c2c334696905-kube-api-access-45zv7" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751547    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae0f3040-049e-4f4e-a6e5-206b31aeea55" volumeName="kubernetes.io/projected/ae0f3040-049e-4f4e-a6e5-206b31aeea55-kube-api-access-jpqz6" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751561    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc3514a7-8b72-40ff-b45a-1c2005b59360" volumeName="kubernetes.io/projected/fc3514a7-8b72-40ff-b45a-1c2005b59360-kube-api-access-cpx52" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751575    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="433f04c0-a9ea-45c8-a2b7-245980607391" volumeName="kubernetes.io/projected/433f04c0-a9ea-45c8-a2b7-245980607391-kube-api-access-4q94d" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751587    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="794aa402-53bf-43fc-b21b-a176bbbcd132" volumeName="kubernetes.io/configmap/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-config" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751600    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d95535e9-e677-463a-a662-7e8045c71d9b" volumeName="kubernetes.io/projected/d95535e9-e677-463a-a662-7e8045c71d9b-kube-api-access-78557" seLinuxMountContext=""
Oct 30 00:41:27.751609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751612    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d95535e9-e677-463a-a662-7e8045c71d9b" volumeName="kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-stats-auth" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751625    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493" volumeName="kubernetes.io/projected/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-kube-api-access-kksvb" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751649    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ac86565-2bfb-45e4-840d-c0f3a29cdfa5" volumeName="kubernetes.io/configmap/2ac86565-2bfb-45e4-840d-c0f3a29cdfa5-konnectivity-ca" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751661    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c48ee8c6-60b2-4229-a013-4199c44813e4" volumeName="kubernetes.io/configmap/c48ee8c6-60b2-4229-a013-4199c44813e4-trusted-ca" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751674    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19a807d5-c007-47b4-953c-28de606ababc" volumeName="kubernetes.io/configmap/19a807d5-c007-47b4-953c-28de606ababc-serviceca" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751689    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19a807d5-c007-47b4-953c-28de606ababc" volumeName="kubernetes.io/projected/19a807d5-c007-47b4-953c-28de606ababc-kube-api-access-8nkxz" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751702    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="433f04c0-a9ea-45c8-a2b7-245980607391" volumeName="kubernetes.io/secret/433f04c0-a9ea-45c8-a2b7-245980607391-serving-cert" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751727    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c48ee8c6-60b2-4229-a013-4199c44813e4" volumeName="kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-kube-api-access-g77n6" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751741    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cee8ca5c-ba97-47b5-b22f-2e161bb93fdc" volumeName="kubernetes.io/configmap/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-config-volume" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751750    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cee8ca5c-ba97-47b5-b22f-2e161bb93fdc" volumeName="kubernetes.io/empty-dir/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-tmp-dir" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751757    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2dadf26-b255-4281-91f4-98536d370d21" volumeName="kubernetes.io/projected/a2dadf26-b255-4281-91f4-98536d370d21-kube-api-access-s72cg" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751765    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="380df728-1bf4-4258-ba04-c31e1d28a3aa" volumeName="kubernetes.io/configmap/380df728-1bf4-4258-ba04-c31e1d28a3aa-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751772    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="380df728-1bf4-4258-ba04-c31e1d28a3aa" volumeName="kubernetes.io/projected/380df728-1bf4-4258-ba04-c31e1d28a3aa-kube-api-access-j89wv" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751780    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4770276e-e9dc-4f6c-94f6-d2dfe2f6f115" volumeName="kubernetes.io/empty-dir/4770276e-e9dc-4f6c-94f6-d2dfe2f6f115-tmp-dir" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751787    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="564f9ba3-d19e-4cac-9021-81939eb8e631" volumeName="kubernetes.io/configmap/564f9ba3-d19e-4cac-9021-81939eb8e631-trusted-ca" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751795    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c48ee8c6-60b2-4229-a013-4199c44813e4" volumeName="kubernetes.io/secret/c48ee8c6-60b2-4229-a013-4199c44813e4-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751802    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0142f4ce-76c5-4ca1-ac00-c2c334696905" volumeName="kubernetes.io/configmap/0142f4ce-76c5-4ca1-ac00-c2c334696905-service-ca-bundle" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751810    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc" volumeName="kubernetes.io/projected/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-kube-api-access-v7glf" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751817    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59e5f12b-4a3e-449b-b285-17747780e1cf" volumeName="kubernetes.io/projected/59e5f12b-4a3e-449b-b285-17747780e1cf-kube-api-access-cmm4r" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751824    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55" volumeName="kubernetes.io/secret/a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55-cert" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751831    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c75bc9c9-74be-412b-a5ac-794f971bea31" volumeName="kubernetes.io/configmap/c75bc9c9-74be-412b-a5ac-794f971bea31-iptables-alerter-script" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751838    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="380df728-1bf4-4258-ba04-c31e1d28a3aa" volumeName="kubernetes.io/configmap/380df728-1bf4-4258-ba04-c31e1d28a3aa-ovnkube-config" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751849    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="564f9ba3-d19e-4cac-9021-81939eb8e631" volumeName="kubernetes.io/secret/564f9ba3-d19e-4cac-9021-81939eb8e631-installation-pull-secrets" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751861    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a145873-a56a-4fe8-bdb5-dd0e6837b948" volumeName="kubernetes.io/configmap/6a145873-a56a-4fe8-bdb5-dd0e6837b948-nginx-conf" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751869    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e4dbba6-28af-46b3-ac90-37ce437fd318" volumeName="kubernetes.io/configmap/8e4dbba6-28af-46b3-ac90-37ce437fd318-config" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751875    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbef968a-7aa4-4e23-9278-3706e60d8c0b" volumeName="kubernetes.io/secret/bbef968a-7aa4-4e23-9278-3706e60d8c0b-original-pull-secret" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751883    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c48ee8c6-60b2-4229-a013-4199c44813e4" volumeName="kubernetes.io/empty-dir/c48ee8c6-60b2-4229-a013-4199c44813e4-ca-trust-extracted" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751890    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d95535e9-e677-463a-a662-7e8045c71d9b" volumeName="kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-default-certificate" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751899    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0142f4ce-76c5-4ca1-ac00-c2c334696905" volumeName="kubernetes.io/empty-dir/0142f4ce-76c5-4ca1-ac00-c2c334696905-tmp" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751909    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0eac4baa-d0d9-43c0-8f81-3733a026e091" volumeName="kubernetes.io/configmap/0eac4baa-d0d9-43c0-8f81-3733a026e091-multus-daemon-config" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751916    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="564f9ba3-d19e-4cac-9021-81939eb8e631" volumeName="kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-bound-sa-token" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751924    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="794aa402-53bf-43fc-b21b-a176bbbcd132" volumeName="kubernetes.io/projected/794aa402-53bf-43fc-b21b-a176bbbcd132-kube-api-access-79mmb" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751932    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c48ee8c6-60b2-4229-a013-4199c44813e4" volumeName="kubernetes.io/configmap/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-certificates" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751939    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e05b9b53-5f5b-402e-8599-ce61481346e6" volumeName="kubernetes.io/projected/e05b9b53-5f5b-402e-8599-ce61481346e6-kube-api-access-l8xgb" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751947    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5fade25-18f0-4e3f-978a-5bb39a356ab0" volumeName="kubernetes.io/projected/f5fade25-18f0-4e3f-978a-5bb39a356ab0-kube-api-access-tk7mm" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751955    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0142f4ce-76c5-4ca1-ac00-c2c334696905" volumeName="kubernetes.io/empty-dir/0142f4ce-76c5-4ca1-ac00-c2c334696905-snapshots" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751962    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ac86565-2bfb-45e4-840d-c0f3a29cdfa5" volumeName="kubernetes.io/secret/2ac86565-2bfb-45e4-840d-c0f3a29cdfa5-agent-certs" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751969    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="380df728-1bf4-4258-ba04-c31e1d28a3aa" volumeName="kubernetes.io/secret/380df728-1bf4-4258-ba04-c31e1d28a3aa-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751977    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae0f3040-049e-4f4e-a6e5-206b31aeea55" volumeName="kubernetes.io/empty-dir/ae0f3040-049e-4f4e-a6e5-206b31aeea55-tmp" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751984    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c48ee8c6-60b2-4229-a013-4199c44813e4" volumeName="kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-bound-sa-token" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.751998    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0eac4baa-d0d9-43c0-8f81-3733a026e091" volumeName="kubernetes.io/projected/0eac4baa-d0d9-43c0-8f81-3733a026e091-kube-api-access-nnf9q" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.752006    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="564f9ba3-d19e-4cac-9021-81939eb8e631" volumeName="kubernetes.io/empty-dir/564f9ba3-d19e-4cac-9021-81939eb8e631-ca-trust-extracted" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.752013    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5fade25-18f0-4e3f-978a-5bb39a356ab0" volumeName="kubernetes.io/configmap/f5fade25-18f0-4e3f-978a-5bb39a356ab0-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.752020    4864 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="380df728-1bf4-4258-ba04-c31e1d28a3aa" volumeName="kubernetes.io/configmap/380df728-1bf4-4258-ba04-c31e1d28a3aa-env-overrides" seLinuxMountContext=""
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.752028    4864 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 00:41:27.752371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.752033    4864 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 00:41:27.753688 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.752608    4864 factory.go:153] Registering CRI-O factory
Oct 30 00:41:27.753688 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.752621    4864 factory.go:223] Registration of the crio container factory successfully
Oct 30 00:41:27.753688 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.752655    4864 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 00:41:27.753688 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.752662    4864 factory.go:55] Registering systemd factory
Oct 30 00:41:27.753688 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.752667    4864 factory.go:223] Registration of the systemd container factory successfully
Oct 30 00:41:27.753688 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.752678    4864 factory.go:103] Registering Raw factory
Oct 30 00:41:27.753688 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.752687    4864 manager.go:1199] Started watching for new ooms in manager
Oct 30 00:41:27.753688 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.752911    4864 manager.go:319] Starting recovery of all containers
Oct 30 00:41:27.771995 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.771970    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/1.log"
Oct 30 00:41:27.771995 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.771971    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/1.log"
Oct 30 00:41:27.783441 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.783399    4864 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 00:41:27.785168 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.785132    4864 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 00:41:27.785168 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.785164    4864 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 00:41:27.785327 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.785189    4864 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 00:41:27.785327 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:27.785230    4864 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 00:41:27.790151 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.790047    4864 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:41:27.790279 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.790244    4864 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 00:41:27.791115 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.791067    4864 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:41:27.793089 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.793058    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/1.log"
Oct 30 00:41:27.793190 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.793095    4864 generic.go:356] "Generic (PLEG): container finished" podID="8e4dbba6-28af-46b3-ac90-37ce437fd318" containerID="a31b42b1c0dd5bc05a1170b6f3ab972f96c72f863fca50fdd3997fb3c7fd610b" exitCode=255
Oct 30 00:41:27.803947 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.802022    4864 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:41:27.803947 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.802238    4864 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:41:27.810169 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.810139    4864 generic.go:356] "Generic (PLEG): container finished" podID="e4c9778b041df41465cbcba4cba59e17" containerID="78b07aba50b7255e46ab7007879cec55a1150e1b3d4a74927949cc6efb7ead09" exitCode=0
Oct 30 00:41:27.825382 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.825353    4864 generic.go:356] "Generic (PLEG): container finished" podID="f5fade25-18f0-4e3f-978a-5bb39a356ab0" containerID="abb929bbd3d85b3408431b1db469efaae4b06a19c4e58a22651f45ac5ee053b4" exitCode=0
Oct 30 00:41:27.825382 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.825381    4864 generic.go:356] "Generic (PLEG): container finished" podID="f5fade25-18f0-4e3f-978a-5bb39a356ab0" containerID="34700786c0ee9bd743338750d8da2f7d262537fd94d2f53b12ca49d79c96c06d" exitCode=0
Oct 30 00:41:27.825529 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.825388    4864 generic.go:356] "Generic (PLEG): container finished" podID="f5fade25-18f0-4e3f-978a-5bb39a356ab0" containerID="9519e32db0c84191cff88665ef3b933d8e0907e7bb53786d97768848c301eb9f" exitCode=0
Oct 30 00:41:27.825529 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.825394    4864 generic.go:356] "Generic (PLEG): container finished" podID="f5fade25-18f0-4e3f-978a-5bb39a356ab0" containerID="01a246586e0cc9a138277d595d7a76b254390f6c1d7f06c71e6d177180e6fa8e" exitCode=0
Oct 30 00:41:27.825529 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.825401    4864 generic.go:356] "Generic (PLEG): container finished" podID="f5fade25-18f0-4e3f-978a-5bb39a356ab0" containerID="8f4fcb280a80b28a33e6695b6feb61747757f015f0a45207ad16b0b5fcb541b7" exitCode=0
Oct 30 00:41:27.825529 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.825408    4864 generic.go:356] "Generic (PLEG): container finished" podID="f5fade25-18f0-4e3f-978a-5bb39a356ab0" containerID="b74eb6e2959251917f1841053f8acb77e46517f2743e58af83e07a9f6a7423e3" exitCode=0
Oct 30 00:41:27.842490 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.842442    4864 manager.go:324] Recovery completed
Oct 30 00:41:27.861442 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.861419    4864 cpu_manager.go:226] "Starting" policy="none"
Oct 30 00:41:27.861442 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.861435    4864 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 00:41:27.861568 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.861452    4864 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 00:41:27.861599 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.861569    4864 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 00:41:27.861599 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.861580    4864 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 00:41:27.861599 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.861596    4864 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 00:41:27.861687 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.861602    4864 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 00:41:27.861687 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.861607    4864 policy_none.go:50] "Start"
Oct 30 00:41:27.861687 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.861620    4864 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 00:41:27.861687 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.861629    4864 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 00:41:27.861793 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.861749    4864 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 00:41:27.861793 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.861756    4864 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 00:41:27.861793 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.861761    4864 policy_none.go:44] "Start"
Oct 30 00:41:27.865942 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.865914    4864 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 00:41:27.868522 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.868506    4864 manager.go:341] "Starting Device Plugin manager"
Oct 30 00:41:27.869025 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:27.868533    4864 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 00:41:27.869124 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.869029    4864 server.go:87] "Starting device plugin registration server"
Oct 30 00:41:27.869212 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.869202    4864 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 00:41:27.869247 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.869213    4864 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 00:41:27.869337 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.869316    4864 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 00:41:27.869491 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.869456    4864 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 00:41:27.869491 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.869491    4864 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 00:41:27.869581 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.869499    4864 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 00:41:27.869609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.869540    4864 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:41:27.869480105Z","UUID":"bee7e38d-da08-4f49-bd23-cbcc5030673b","Handler":null,"Name":"","Endpoint":""}
Oct 30 00:41:27.871152 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:27.871130    4864 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 00:41:27.871481 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.871459    4864 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 00:41:27.871566 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.871491    4864 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 00:41:27.885375 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.885321    4864 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-210.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal"]
Oct 30 00:41:27.886035 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.886010    4864 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-210.ec2.internal"
Oct 30 00:41:27.886171 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.886052    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-210.ec2.internal" event={"ID":"17edaf9736d7ee1e0409a6d573ccb434","Type":"ContainerStarted","Data":"6f7a116041a25cf213d0e932612dd9d753549f37fbc586e07552b75c98efb608"}
Oct 30 00:41:27.886171 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.886109    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-210.ec2.internal" event={"ID":"17edaf9736d7ee1e0409a6d573ccb434","Type":"ContainerStarted","Data":"e00c480be6b0794693cf776efc17b9266d72737419c9ab17343e8f19abe31439"}
Oct 30 00:41:27.886171 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.886128    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal" event={"ID":"e4c9778b041df41465cbcba4cba59e17","Type":"ContainerStarted","Data":"a16c7065be591119eaac96e1e8818460066d2771465d4fd346e7a0d702cc8912"}
Oct 30 00:41:27.886171 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.886146    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal" event={"ID":"e4c9778b041df41465cbcba4cba59e17","Type":"ContainerDied","Data":"78b07aba50b7255e46ab7007879cec55a1150e1b3d4a74927949cc6efb7ead09"}
Oct 30 00:41:27.886407 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.886174    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal" event={"ID":"e4c9778b041df41465cbcba4cba59e17","Type":"ContainerStarted","Data":"6ff453ef83e8b4374463dbceb3eda86c51c7b10b2b23e81a7aabffa9367d65c2"}
Oct 30 00:41:27.886407 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.886322    4864 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal"
Oct 30 00:41:27.909017 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.908988    4864 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:41:27.909157 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:27.909043    4864 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-210.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-210.ec2.internal"
Oct 30 00:41:27.909338 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.909323    4864 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:41:27.909389 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:27.909361    4864 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal"
Oct 30 00:41:27.969805 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.969781    4864 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:41:27.970987 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.970970    4864 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:41:27.971093 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.970998    4864 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:41:27.971093 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.971011    4864 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-210.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:41:27.971093 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.971056    4864 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-210.ec2.internal"
Oct 30 00:41:27.990714 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.990691    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17edaf9736d7ee1e0409a6d573ccb434-config\") pod \"kube-apiserver-proxy-ip-10-0-134-210.ec2.internal\" (UID: \"17edaf9736d7ee1e0409a6d573ccb434\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-210.ec2.internal"
Oct 30 00:41:27.990782 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.990718    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e4c9778b041df41465cbcba4cba59e17-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal\" (UID: \"e4c9778b041df41465cbcba4cba59e17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal"
Oct 30 00:41:27.990782 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:27.990737    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4c9778b041df41465cbcba4cba59e17-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal\" (UID: \"e4c9778b041df41465cbcba4cba59e17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal"
Oct 30 00:41:28.000558 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.000538    4864 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-210.ec2.internal"
Oct 30 00:41:28.000623 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.000604    4864 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-210.ec2.internal"
Oct 30 00:41:28.091261 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.091224    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17edaf9736d7ee1e0409a6d573ccb434-config\") pod \"kube-apiserver-proxy-ip-10-0-134-210.ec2.internal\" (UID: \"17edaf9736d7ee1e0409a6d573ccb434\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-210.ec2.internal"
Oct 30 00:41:28.091261 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.091261    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e4c9778b041df41465cbcba4cba59e17-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal\" (UID: \"e4c9778b041df41465cbcba4cba59e17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal"
Oct 30 00:41:28.091435 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.091280    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4c9778b041df41465cbcba4cba59e17-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal\" (UID: \"e4c9778b041df41465cbcba4cba59e17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal"
Oct 30 00:41:28.091435 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.091335    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4c9778b041df41465cbcba4cba59e17-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal\" (UID: \"e4c9778b041df41465cbcba4cba59e17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal"
Oct 30 00:41:28.091435 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.091336    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17edaf9736d7ee1e0409a6d573ccb434-config\") pod \"kube-apiserver-proxy-ip-10-0-134-210.ec2.internal\" (UID: \"17edaf9736d7ee1e0409a6d573ccb434\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-210.ec2.internal"
Oct 30 00:41:28.091435 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.091336    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e4c9778b041df41465cbcba4cba59e17-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal\" (UID: \"e4c9778b041df41465cbcba4cba59e17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal"
Oct 30 00:41:28.736635 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.736602    4864 apiserver.go:52] "Watching apiserver"
Oct 30 00:41:28.781926 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.781893    4864 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 00:41:28.782636 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.782611    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hf7tk","openshift-network-diagnostics/network-check-source-6f78d59f99-9zmkk","openshift-image-registry/image-registry-5f45784fb5-r5j44","openshift-image-registry/node-ca-qdjmq","kube-system/kube-apiserver-proxy-ip-10-0-134-210.ec2.internal","openshift-insights/insights-runtime-extractor-f9vrm","kube-system/global-pull-secret-syncer-wk65n","openshift-dns/node-resolver-49rdp","kube-system/kas-connection-checker-685895c7c9-pq5xl","openshift-multus/network-metrics-daemon-b4686","openshift-network-console/networking-console-plugin-86c478bb88-j7px8","openshift-insights/insights-operator-7c4986d449-xxvch","openshift-multus/multus-additional-cni-plugins-tz5vn","openshift-cluster-node-tuning-operator/tuned-q4bch","openshift-image-registry/image-registry-654dd5d568-jdc79","openshift-ingress/router-default-94c9fcc47-fdtjt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn","openshift-network-diagnostics/network-check-target-jh728","kube-system/konnectivity-agent-77dcb","openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl","openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s","openshift-multus/multus-qp98l","kube-system/kas-connection-checker-685895c7c9-dtn28","openshift-dns/dns-default-wq8t9","kube-system/kas-connection-checker-685895c7c9-6q8k5","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs","openshift-ovn-kubernetes/ovnkube-node-7855k","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nkb8p","openshift-console-operator/console-operator-7cd56b8dbc-jk65g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal","openshift-network-operator/iptables-alerter-ljlrc","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"]
Oct 30 00:41:28.785292 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.785272    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:28.788668 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.788601    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:28.788773 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.788670    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:28.788828 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.788781    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"
Oct 30 00:41:28.789924 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.789902    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-j7px8"
Oct 30 00:41:28.792207 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.792190    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:41:28.794002 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.793984    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:28.796538 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.796521    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:28.797031 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.797015    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:28.806086 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.806048    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 00:41:28.806186 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.806101    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 00:41:28.806375 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.806360    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 00:41:28.821894 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.821873    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 00:41:28.821971 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.821906    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 00:41:28.831880 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.831854    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 00:41:28.831949 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.831901    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 00:41:28.832335 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.832314    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-92qkq\""
Oct 30 00:41:28.832421 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.832344    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 00:41:28.832526 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.832511    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 00:41:28.832584 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.832540    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 00:41:28.832584 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.832567    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 00:41:28.832657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.832610    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-szv55\""
Oct 30 00:41:28.832712 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.832694    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f5gxs\""
Oct 30 00:41:28.832765 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.832754    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 00:41:28.832810 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.832793    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 00:41:28.832810 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.832797    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 00:41:28.832901 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.832888    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 00:41:28.833100 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.833087    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-lxq9t\""
Oct 30 00:41:28.833137 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.833112    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wm8np\""
Oct 30 00:41:28.833248 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.833236    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7sn5q\""
Oct 30 00:41:28.833408 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.833396    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5b9kk\""
Oct 30 00:41:28.833483 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.833469    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jgsn4\""
Oct 30 00:41:28.833562 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.833549    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cg6zp\""
Oct 30 00:41:28.833600 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.833554    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-z4qxk\""
Oct 30 00:41:28.833735 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.833723    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 00:41:28.833790 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.833776    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6kdk2\""
Oct 30 00:41:28.834005 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.833992    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 00:41:28.834048 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.833998    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 00:41:28.834048 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.834028    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7htjs\""
Oct 30 00:41:28.834048 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.834046    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z5p7p\""
Oct 30 00:41:28.834179 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.834059    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 00:41:28.834179 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.834089    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 00:41:28.834179 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.834108    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-x7m5x\""
Oct 30 00:41:28.834179 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.834149    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wwjm6\""
Oct 30 00:41:28.834298 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.834183    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l8pvs\""
Oct 30 00:41:28.838375 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.838359    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qs9gb\""
Oct 30 00:41:28.838516 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.838496    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 00:41:28.838953 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.838937    4864 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal"
Oct 30 00:41:28.839706 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.839690    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 00:41:28.840050 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.840034    4864 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-210.ec2.internal"
Oct 30 00:41:28.843164 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.843146    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 00:41:28.850840 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.850823    4864 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 00:41:28.859773 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.859747    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 00:41:28.860110 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860095    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 00:41:28.860201 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860139    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 00:41:28.860295 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860273    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 00:41:28.860352 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860295    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 00:41:28.860352 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860269    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 00:41:28.860352 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860305    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 00:41:28.860352 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860337    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 00:41:28.860352 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860344    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 00:41:28.860352 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860244    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 00:41:28.860619 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860354    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 00:41:28.860619 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860283    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 00:41:28.860619 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860372    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 00:41:28.860619 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860382    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 00:41:28.860619 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860420    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:41:28.860619 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860510    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 00:41:28.860619 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860515    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 00:41:28.860619 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860512    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 00:41:28.860619 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860534    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 00:41:28.860619 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860538    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 00:41:28.860619 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860543    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 00:41:28.860619 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860545    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 00:41:28.860619 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860563    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:41:28.860619 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860568    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 00:41:28.860619 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860568    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 00:41:28.860619 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860584    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 00:41:28.860619 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860581    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 00:41:28.860619 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860592    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 00:41:28.860619 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860603    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:41:28.860619 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860609    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 00:41:28.860619 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860620    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 00:41:28.860619 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860624    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 00:41:28.861427 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860678    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 00:41:28.861427 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860690    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:41:28.861427 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860811    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:41:28.861427 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860895    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 00:41:28.861427 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860940    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 00:41:28.861427 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.860997    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 00:41:28.861427 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.861050    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 00:41:28.861427 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.861059    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 00:41:28.861427 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.861194    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:41:28.861427 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.861222    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 00:41:28.861427 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.861263    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 00:41:28.861427 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.861297    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 00:41:28.861427 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.861343    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 00:41:28.861427 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.861375    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 00:41:28.861938 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.861482    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 00:41:28.861938 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.861602    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 00:41:28.861938 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.861900    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 00:41:28.870889 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.870874    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 00:41:28.872550 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.872531    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 00:41:28.872667 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.872649    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 00:41:28.872742 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.872672    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 00:41:28.872742 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.872697    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:41:28.872742 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.872705    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 00:41:28.872742 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.872717    4864 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:41:28.872960 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.872757    4864 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:41:28.872960 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:28.872765    4864 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-210.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-210.ec2.internal"
Oct 30 00:41:28.872960 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:28.872792    4864 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-210.ec2.internal"
Oct 30 00:41:28.873127 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.873111    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 00:41:28.873217 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.873203    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 00:41:28.894489 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.894459    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7glf\" (UniqueName: \"kubernetes.io/projected/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-kube-api-access-v7glf\") pod \"cluster-monitoring-operator-884576c57-lfnrl\" (UID: \"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:28.894597 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.894493    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e4dbba6-28af-46b3-ac90-37ce437fd318-config\") pod \"console-operator-7cd56b8dbc-jk65g\" (UID: \"8e4dbba6-28af-46b3-ac90-37ce437fd318\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:28.894597 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.894524    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-modprobe-d\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:28.894597 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.894548    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-lib-modules\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:28.894597 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.894573    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79mmb\" (UniqueName: \"kubernetes.io/projected/794aa402-53bf-43fc-b21b-a176bbbcd132-kube-api-access-79mmb\") pod \"cluster-samples-operator-6ddb6c8b94-gh5xc\" (UID: \"794aa402-53bf-43fc-b21b-a176bbbcd132\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"
Oct 30 00:41:28.894778 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.894610    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c48ee8c6-60b2-4229-a013-4199c44813e4-installation-pull-secrets\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:28.894778 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.894720    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e4dbba6-28af-46b3-ac90-37ce437fd318-serving-cert\") pod \"console-operator-7cd56b8dbc-jk65g\" (UID: \"8e4dbba6-28af-46b3-ac90-37ce437fd318\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:28.894778 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.894751    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-device-dir\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:41:28.894913 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.894778    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpn92\" (UniqueName: \"kubernetes.io/projected/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-kube-api-access-bpn92\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:41:28.894913 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.894802    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e4dbba6-28af-46b3-ac90-37ce437fd318-config\") pod \"console-operator-7cd56b8dbc-jk65g\" (UID: \"8e4dbba6-28af-46b3-ac90-37ce437fd318\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:28.894913 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.894819    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-multus-cni-dir\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:28.894913 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.894879    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0eac4baa-d0d9-43c0-8f81-3733a026e091-cni-binary-copy\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:28.894913 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.894904    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c48ee8c6-60b2-4229-a013-4199c44813e4-installation-pull-secrets\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:28.895138 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.894906    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-sys\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:28.895138 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.894943    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/564f9ba3-d19e-4cac-9021-81939eb8e631-installation-pull-secrets\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:28.895138 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.894975    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-cni-bin\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.895138 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.894983    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e4dbba6-28af-46b3-ac90-37ce437fd318-serving-cert\") pod \"console-operator-7cd56b8dbc-jk65g\" (UID: \"8e4dbba6-28af-46b3-ac90-37ce437fd318\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:28.895138 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895023    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-sys-fs\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:41:28.895293 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895161    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-run-multus-certs\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:28.895293 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895186    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:28.895293 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895211    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5fade25-18f0-4e3f-978a-5bb39a356ab0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:41:28.895293 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895220    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0eac4baa-d0d9-43c0-8f81-3733a026e091-cni-binary-copy\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:28.895293 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895234    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8e4dbba6-28af-46b3-ac90-37ce437fd318-trusted-ca\") pod \"console-operator-7cd56b8dbc-jk65g\" (UID: \"8e4dbba6-28af-46b3-ac90-37ce437fd318\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:28.895293 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895236    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/564f9ba3-d19e-4cac-9021-81939eb8e631-installation-pull-secrets\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:28.895293 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895274    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0142f4ce-76c5-4ca1-ac00-c2c334696905-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:28.895591 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895363    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-run-ovn-kubernetes\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.895591 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895400    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-tuned\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:28.895591 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895485    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-tmp-dir\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:28.895591 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895503    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-tuned\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:28.895591 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895511    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8e4dbba6-28af-46b3-ac90-37ce437fd318-trusted-ca\") pod \"console-operator-7cd56b8dbc-jk65g\" (UID: \"8e4dbba6-28af-46b3-ac90-37ce437fd318\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:28.895591 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895534    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0142f4ce-76c5-4ca1-ac00-c2c334696905-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:28.895591 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895537    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-etc-kubernetes\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:28.895591 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895582    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbef968a-7aa4-4e23-9278-3706e60d8c0b-dbus\") pod \"global-pull-secret-syncer-wk65n\" (UID: \"bbef968a-7aa4-4e23-9278-3706e60d8c0b\") " pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:41:28.895591 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895560    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-tmp-dir\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:28.895950 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895600    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-systemd-units\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.895950 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895640    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gh5xc\" (UID: \"794aa402-53bf-43fc-b21b-a176bbbcd132\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"
Oct 30 00:41:28.895950 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895661    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmm4r\" (UniqueName: \"kubernetes.io/projected/59e5f12b-4a3e-449b-b285-17747780e1cf-kube-api-access-cmm4r\") pod \"service-ca-operator-544d749c8c-jbp7s\" (UID: \"59e5f12b-4a3e-449b-b285-17747780e1cf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s"
Oct 30 00:41:28.895950 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895676    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-lfnrl\" (UID: \"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:28.895950 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895712    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45zv7\" (UniqueName: \"kubernetes.io/projected/0142f4ce-76c5-4ca1-ac00-c2c334696905-kube-api-access-45zv7\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:28.895950 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895729    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-multus-socket-dir-parent\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:28.895950 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895747    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-run-openvswitch\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.895950 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895774    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/380df728-1bf4-4258-ba04-c31e1d28a3aa-ovn-node-metrics-cert\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.895950 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895806    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a50fdaf-7855-4033-96c8-6cb23212a986-crio-socket\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:28.895950 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895836    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae0f3040-049e-4f4e-a6e5-206b31aeea55-tmp\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:28.895950 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895870    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s72cg\" (UniqueName: \"kubernetes.io/projected/a2dadf26-b255-4281-91f4-98536d370d21-kube-api-access-s72cg\") pod \"kas-connection-checker-685895c7c9-dtn28\" (UID: \"a2dadf26-b255-4281-91f4-98536d370d21\") " pod="kube-system/kas-connection-checker-685895c7c9-dtn28"
Oct 30 00:41:28.895950 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895896    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbef968a-7aa4-4e23-9278-3706e60d8c0b-kubelet-config\") pod \"global-pull-secret-syncer-wk65n\" (UID: \"bbef968a-7aa4-4e23-9278-3706e60d8c0b\") " pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:41:28.895950 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895920    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/19a807d5-c007-47b4-953c-28de606ababc-serviceca\") pod \"node-ca-qdjmq\" (UID: \"19a807d5-c007-47b4-953c-28de606ababc\") " pod="openshift-image-registry/node-ca-qdjmq"
Oct 30 00:41:28.896501 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895964    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-kubelet\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.896501 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.895969    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae0f3040-049e-4f4e-a6e5-206b31aeea55-tmp\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:28.896501 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896025    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-var-lib-openvswitch\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.896501 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896035    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-lfnrl\" (UID: \"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:28.896501 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896044    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/380df728-1bf4-4258-ba04-c31e1d28a3aa-ovn-node-metrics-cert\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.896501 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896083    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c48ee8c6-60b2-4229-a013-4199c44813e4-ca-trust-extracted\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:28.896501 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896132    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/19a807d5-c007-47b4-953c-28de606ababc-serviceca\") pod \"node-ca-qdjmq\" (UID: \"19a807d5-c007-47b4-953c-28de606ababc\") " pod="openshift-image-registry/node-ca-qdjmq"
Oct 30 00:41:28.896501 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896149    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a50fdaf-7855-4033-96c8-6cb23212a986-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:28.896501 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896160    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c48ee8c6-60b2-4229-a013-4199c44813e4-ca-trust-extracted\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:28.896501 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896177    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8s6t\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-kube-api-access-d8s6t\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:28.896501 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896216    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0142f4ce-76c5-4ca1-ac00-c2c334696905-snapshots\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:28.896501 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896263    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-var-lib-cni-multus\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:28.896501 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896290    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0142f4ce-76c5-4ca1-ac00-c2c334696905-snapshots\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:28.896501 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896303    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ac86565-2bfb-45e4-840d-c0f3a29cdfa5-agent-certs\") pod \"konnectivity-agent-77dcb\" (UID: \"2ac86565-2bfb-45e4-840d-c0f3a29cdfa5\") " pod="kube-system/konnectivity-agent-77dcb"
Oct 30 00:41:28.896501 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896450    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c75bc9c9-74be-412b-a5ac-794f971bea31-host-slash\") pod \"iptables-alerter-ljlrc\" (UID: \"c75bc9c9-74be-412b-a5ac-794f971bea31\") " pod="openshift-network-operator/iptables-alerter-ljlrc"
Oct 30 00:41:28.896501 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896484    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/564f9ba3-d19e-4cac-9021-81939eb8e631-image-registry-private-configuration\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:28.896501 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896508    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:28.897227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896526    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0142f4ce-76c5-4ca1-ac00-c2c334696905-tmp\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:28.897227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896542    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-system-cni-dir\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:28.897227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896599    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-systemd\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:28.897227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896639    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0142f4ce-76c5-4ca1-ac00-c2c334696905-tmp\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:28.897227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896639    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ac86565-2bfb-45e4-840d-c0f3a29cdfa5-agent-certs\") pod \"konnectivity-agent-77dcb\" (UID: \"2ac86565-2bfb-45e4-840d-c0f3a29cdfa5\") " pod="kube-system/konnectivity-agent-77dcb"
Oct 30 00:41:28.897227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896659    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-run\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:28.897227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896686    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/564f9ba3-d19e-4cac-9021-81939eb8e631-image-registry-private-configuration\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:28.897227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896702    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:28.897227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896731    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-var-lib-cni-bin\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:28.897227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896780    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-multus-conf-dir\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:28.897227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896805    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ac86565-2bfb-45e4-840d-c0f3a29cdfa5-konnectivity-ca\") pod \"konnectivity-agent-77dcb\" (UID: \"2ac86565-2bfb-45e4-840d-c0f3a29cdfa5\") " pod="kube-system/konnectivity-agent-77dcb"
Oct 30 00:41:28.897227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896822    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f5fade25-18f0-4e3f-978a-5bb39a356ab0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:41:28.897227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896852    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-socket-dir\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:41:28.897227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896894    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/433f04c0-a9ea-45c8-a2b7-245980607391-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs\" (UID: \"433f04c0-a9ea-45c8-a2b7-245980607391\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs"
Oct 30 00:41:28.897227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.896922    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-etc-openvswitch\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.897227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897003    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-sysctl-conf\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:28.897227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897024    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ac86565-2bfb-45e4-840d-c0f3a29cdfa5-konnectivity-ca\") pod \"konnectivity-agent-77dcb\" (UID: \"2ac86565-2bfb-45e4-840d-c0f3a29cdfa5\") " pod="kube-system/konnectivity-agent-77dcb"
Oct 30 00:41:28.897227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897045    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxvpz\" (UniqueName: \"kubernetes.io/projected/0a50fdaf-7855-4033-96c8-6cb23212a986-kube-api-access-pxvpz\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:28.897227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897105    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5fade25-18f0-4e3f-978a-5bb39a356ab0-cnibin\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:41:28.897227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897111    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/433f04c0-a9ea-45c8-a2b7-245980607391-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs\" (UID: \"433f04c0-a9ea-45c8-a2b7-245980607391\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs"
Oct 30 00:41:28.897227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897182    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f5fade25-18f0-4e3f-978a-5bb39a356ab0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:41:28.897227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897213    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tkrk\" (UniqueName: \"kubernetes.io/projected/4509811e-30a0-4c8b-b21e-58826bd1f6c6-kube-api-access-5tkrk\") pod \"kas-connection-checker-685895c7c9-6q8k5\" (UID: \"4509811e-30a0-4c8b-b21e-58826bd1f6c6\") " pod="kube-system/kas-connection-checker-685895c7c9-6q8k5"
Oct 30 00:41:28.898260 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897245    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/59e5f12b-4a3e-449b-b285-17747780e1cf-config\") pod \"service-ca-operator-544d749c8c-jbp7s\" (UID: \"59e5f12b-4a3e-449b-b285-17747780e1cf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s"
Oct 30 00:41:28.898260 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897266    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-sysctl-d\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:28.898260 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897283    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78557\" (UniqueName: \"kubernetes.io/projected/d95535e9-e677-463a-a662-7e8045c71d9b-kube-api-access-78557\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:28.898260 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897301    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6a145873-a56a-4fe8-bdb5-dd0e6837b948-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j7px8\" (UID: \"6a145873-a56a-4fe8-bdb5-dd0e6837b948\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j7px8"
Oct 30 00:41:28.898260 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897339    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55-cert\") pod \"ingress-canary-hf7tk\" (UID: \"a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55\") " pod="openshift-ingress-canary/ingress-canary-hf7tk"
Oct 30 00:41:28.898260 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897411    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q94d\" (UniqueName: \"kubernetes.io/projected/433f04c0-a9ea-45c8-a2b7-245980607391-kube-api-access-4q94d\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs\" (UID: \"433f04c0-a9ea-45c8-a2b7-245980607391\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs"
Oct 30 00:41:28.898260 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897419    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/59e5f12b-4a3e-449b-b285-17747780e1cf-config\") pod \"service-ca-operator-544d749c8c-jbp7s\" (UID: \"59e5f12b-4a3e-449b-b285-17747780e1cf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s"
Oct 30 00:41:28.898260 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897439    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j89wv\" (UniqueName: \"kubernetes.io/projected/380df728-1bf4-4258-ba04-c31e1d28a3aa-kube-api-access-j89wv\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.898260 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897497    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c48ee8c6-60b2-4229-a013-4199c44813e4-image-registry-private-configuration\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:28.898260 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897528    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kksvb\" (UniqueName: \"kubernetes.io/projected/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-kube-api-access-kksvb\") pod \"network-metrics-daemon-b4686\" (UID: \"0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493\") " pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:41:28.898260 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897549    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55-cert\") pod \"ingress-canary-hf7tk\" (UID: \"a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55\") " pod="openshift-ingress-canary/ingress-canary-hf7tk"
Oct 30 00:41:28.898260 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897574    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-log-socket\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.898260 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897600    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6a145873-a56a-4fe8-bdb5-dd0e6837b948-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j7px8\" (UID: \"6a145873-a56a-4fe8-bdb5-dd0e6837b948\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j7px8"
Oct 30 00:41:28.898260 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897610    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/380df728-1bf4-4258-ba04-c31e1d28a3aa-ovnkube-config\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.898260 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897691    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g77n6\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-kube-api-access-g77n6\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:28.898260 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897720    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk7mm\" (UniqueName: \"kubernetes.io/projected/f5fade25-18f0-4e3f-978a-5bb39a356ab0-kube-api-access-tk7mm\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:41:28.898260 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897720    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c48ee8c6-60b2-4229-a013-4199c44813e4-image-registry-private-configuration\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:28.898260 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897790    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/380df728-1bf4-4258-ba04-c31e1d28a3aa-ovnkube-config\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.898260 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897909    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/564f9ba3-d19e-4cac-9021-81939eb8e631-ca-trust-extracted\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:28.899034 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.897750    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/564f9ba3-d19e-4cac-9021-81939eb8e631-ca-trust-extracted\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:28.899034 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.898428    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-bound-sa-token\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:28.899034 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.898456    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/433f04c0-a9ea-45c8-a2b7-245980607391-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs\" (UID: \"433f04c0-a9ea-45c8-a2b7-245980607391\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs"
Oct 30 00:41:28.899034 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.898486    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:41:28.899034 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.898511    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0142f4ce-76c5-4ca1-ac00-c2c334696905-service-ca-bundle\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:28.899034 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.898556    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-cnibin\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:28.899034 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.898615    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-cni-netd\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.899034 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.898643    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/433f04c0-a9ea-45c8-a2b7-245980607391-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs\" (UID: \"433f04c0-a9ea-45c8-a2b7-245980607391\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs"
Oct 30 00:41:28.899034 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.898666    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-sysconfig\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:28.899034 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.898755    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0142f4ce-76c5-4ca1-ac00-c2c334696905-service-ca-bundle\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:28.899034 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.898756    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a50fdaf-7855-4033-96c8-6cb23212a986-data-volume\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:28.899034 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.898815    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-stats-auth\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:28.899034 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.898820    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a50fdaf-7855-4033-96c8-6cb23212a986-data-volume\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:28.899034 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.898948    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-config-volume\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:28.899034 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.898982    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54t8p\" (UniqueName: \"kubernetes.io/projected/a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55-kube-api-access-54t8p\") pod \"ingress-canary-hf7tk\" (UID: \"a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55\") " pod="openshift-ingress-canary/ingress-canary-hf7tk"
Oct 30 00:41:28.899034 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.898985    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-stats-auth\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:28.899034 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.898997    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c48ee8c6-60b2-4229-a013-4199c44813e4-trusted-ca\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:28.899708 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899101    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nkxz\" (UniqueName: \"kubernetes.io/projected/19a807d5-c007-47b4-953c-28de606ababc-kube-api-access-8nkxz\") pod \"node-ca-qdjmq\" (UID: \"19a807d5-c007-47b4-953c-28de606ababc\") " pod="openshift-image-registry/node-ca-qdjmq"
Oct 30 00:41:28.899708 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899134    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-config-volume\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:28.899708 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899239    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/380df728-1bf4-4258-ba04-c31e1d28a3aa-ovnkube-script-lib\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.899708 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899373    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-kubernetes\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:28.899708 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899408    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c75bc9c9-74be-412b-a5ac-794f971bea31-iptables-alerter-script\") pod \"iptables-alerter-ljlrc\" (UID: \"c75bc9c9-74be-412b-a5ac-794f971bea31\") " pod="openshift-network-operator/iptables-alerter-ljlrc"
Oct 30 00:41:28.899708 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899435    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:28.899708 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899452    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/380df728-1bf4-4258-ba04-c31e1d28a3aa-ovnkube-script-lib\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.899708 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899489    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c48ee8c6-60b2-4229-a013-4199c44813e4-trusted-ca\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:28.899708 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899533    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-run-netns\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:28.899708 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899559    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-slash\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.899708 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899587    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-default-certificate\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:28.899708 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899612    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c75bc9c9-74be-412b-a5ac-794f971bea31-iptables-alerter-script\") pod \"iptables-alerter-ljlrc\" (UID: \"c75bc9c9-74be-412b-a5ac-794f971bea31\") " pod="openshift-network-operator/iptables-alerter-ljlrc"
Oct 30 00:41:28.899708 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899623    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0eac4baa-d0d9-43c0-8f81-3733a026e091-multus-daemon-config\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:28.899708 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899643    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-run-ovn\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.899708 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899659    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5fade25-18f0-4e3f-978a-5bb39a356ab0-system-cni-dir\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:41:28.899708 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899698    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-etc-selinux\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:41:28.900379 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899753    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-default-certificate\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:28.900379 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899763    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbef968a-7aa4-4e23-9278-3706e60d8c0b-original-pull-secret\") pod \"global-pull-secret-syncer-wk65n\" (UID: \"bbef968a-7aa4-4e23-9278-3706e60d8c0b\") " pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:41:28.900379 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899799    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-run-systemd\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.900379 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899800    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0eac4baa-d0d9-43c0-8f81-3733a026e091-multus-daemon-config\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:28.900379 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899861    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r64gt\" (UniqueName: \"kubernetes.io/projected/76b2bcb8-636c-46b5-8dfb-95831470349e-kube-api-access-r64gt\") pod \"network-check-source-6f78d59f99-9zmkk\" (UID: \"76b2bcb8-636c-46b5-8dfb-95831470349e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9zmkk"
Oct 30 00:41:28.900379 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899887    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-var-lib-kubelet\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:28.900379 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899903    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-certificates\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:28.900379 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899952    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbef968a-7aa4-4e23-9278-3706e60d8c0b-original-pull-secret\") pod \"global-pull-secret-syncer-wk65n\" (UID: \"bbef968a-7aa4-4e23-9278-3706e60d8c0b\") " pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:41:28.900379 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899972    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-var-lib-kubelet\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:28.900379 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.899992    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5fade25-18f0-4e3f-978a-5bb39a356ab0-cni-binary-copy\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:41:28.900379 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900013    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/564f9ba3-d19e-4cac-9021-81939eb8e631-trusted-ca\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:28.900379 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900030    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-certificates\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:28.900379 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900043    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnf9q\" (UniqueName: \"kubernetes.io/projected/0eac4baa-d0d9-43c0-8f81-3733a026e091-kube-api-access-nnf9q\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:28.900379 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900101    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19a807d5-c007-47b4-953c-28de606ababc-host\") pod \"node-ca-qdjmq\" (UID: \"19a807d5-c007-47b4-953c-28de606ababc\") " pod="openshift-image-registry/node-ca-qdjmq"
Oct 30 00:41:28.900379 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900128    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwnlf\" (UniqueName: \"kubernetes.io/projected/4770276e-e9dc-4f6c-94f6-d2dfe2f6f115-kube-api-access-lwnlf\") pod \"node-resolver-49rdp\" (UID: \"4770276e-e9dc-4f6c-94f6-d2dfe2f6f115\") " pod="openshift-dns/node-resolver-49rdp"
Oct 30 00:41:28.900379 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900165    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-run-k8s-cni-cncf-io\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:28.900379 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900186    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5fade25-18f0-4e3f-978a-5bb39a356ab0-cni-binary-copy\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:41:28.900379 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900271    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-hostroot\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:28.900379 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900342    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/380df728-1bf4-4258-ba04-c31e1d28a3aa-env-overrides\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.900379 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900382    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq6ls\" (UniqueName: \"kubernetes.io/projected/c75bc9c9-74be-412b-a5ac-794f971bea31-kube-api-access-jq6ls\") pod \"iptables-alerter-ljlrc\" (UID: \"c75bc9c9-74be-412b-a5ac-794f971bea31\") " pod="openshift-network-operator/iptables-alerter-ljlrc"
Oct 30 00:41:28.901099 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900416    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gh5xc\" (UID: \"794aa402-53bf-43fc-b21b-a176bbbcd132\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"
Oct 30 00:41:28.901099 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900448    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpx52\" (UniqueName: \"kubernetes.io/projected/fc3514a7-8b72-40ff-b45a-1c2005b59360-kube-api-access-cpx52\") pod \"kas-connection-checker-685895c7c9-pq5xl\" (UID: \"fc3514a7-8b72-40ff-b45a-1c2005b59360\") " pod="kube-system/kas-connection-checker-685895c7c9-pq5xl"
Oct 30 00:41:28.901099 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900478    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-registration-dir\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:41:28.901099 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900508    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-node-log\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.901099 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900539    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8xgb\" (UniqueName: \"kubernetes.io/projected/e05b9b53-5f5b-402e-8599-ce61481346e6-kube-api-access-l8xgb\") pod \"volume-data-source-validator-58c9f5f46d-nkb8p\" (UID: \"e05b9b53-5f5b-402e-8599-ce61481346e6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nkb8p"
Oct 30 00:41:28.901099 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900538    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/380df728-1bf4-4258-ba04-c31e1d28a3aa-env-overrides\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.901099 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900580    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs\") pod \"network-metrics-daemon-b4686\" (UID: \"0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493\") " pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:41:28.901099 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900635    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4770276e-e9dc-4f6c-94f6-d2dfe2f6f115-hosts-file\") pod \"node-resolver-49rdp\" (UID: \"4770276e-e9dc-4f6c-94f6-d2dfe2f6f115\") " pod="openshift-dns/node-resolver-49rdp"
Oct 30 00:41:28.901099 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900670    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gh5xc\" (UID: \"794aa402-53bf-43fc-b21b-a176bbbcd132\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"
Oct 30 00:41:28.901099 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900693    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:28.901099 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900738    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/59e5f12b-4a3e-449b-b285-17747780e1cf-serving-cert\") pod \"service-ca-operator-544d749c8c-jbp7s\" (UID: \"59e5f12b-4a3e-449b-b285-17747780e1cf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s"
Oct 30 00:41:28.901099 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900783    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lfnrl\" (UID: \"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:28.901099 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900786    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/564f9ba3-d19e-4cac-9021-81939eb8e631-trusted-ca\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:28.901099 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900884    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-run-netns\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.901099 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900894    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/59e5f12b-4a3e-449b-b285-17747780e1cf-serving-cert\") pod \"service-ca-operator-544d749c8c-jbp7s\" (UID: \"59e5f12b-4a3e-449b-b285-17747780e1cf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s"
Oct 30 00:41:28.901099 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900909    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.901099 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900931    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f5fade25-18f0-4e3f-978a-5bb39a356ab0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:41:28.901099 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.900962    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j7px8\" (UID: \"6a145873-a56a-4fe8-bdb5-dd0e6837b948\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j7px8"
Oct 30 00:41:28.901099 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.901058    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sljzz\" (UniqueName: \"kubernetes.io/projected/8e4dbba6-28af-46b3-ac90-37ce437fd318-kube-api-access-sljzz\") pod \"console-operator-7cd56b8dbc-jk65g\" (UID: \"8e4dbba6-28af-46b3-ac90-37ce437fd318\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:28.901099 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.901103    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f5fade25-18f0-4e3f-978a-5bb39a356ab0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:41:28.901099 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.901110    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-bound-sa-token\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:28.901758 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.901139    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-host\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:28.901758 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.901159    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vvj2\" (UniqueName: \"kubernetes.io/projected/204212cc-1c4c-4bfb-8b41-03a98042166c-kube-api-access-7vvj2\") pod \"network-check-target-jh728\" (UID: \"204212cc-1c4c-4bfb-8b41-03a98042166c\") " pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:41:28.901758 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.901182    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75l4z\" (UniqueName: \"kubernetes.io/projected/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-kube-api-access-75l4z\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:28.901758 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.901200    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0142f4ce-76c5-4ca1-ac00-c2c334696905-serving-cert\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:28.901758 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.901215    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpqz6\" (UniqueName: \"kubernetes.io/projected/ae0f3040-049e-4f4e-a6e5-206b31aeea55-kube-api-access-jpqz6\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:28.901758 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.901231    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4770276e-e9dc-4f6c-94f6-d2dfe2f6f115-tmp-dir\") pod \"node-resolver-49rdp\" (UID: \"4770276e-e9dc-4f6c-94f6-d2dfe2f6f115\") " pod="openshift-dns/node-resolver-49rdp"
Oct 30 00:41:28.901758 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.901276    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:28.901758 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.901304    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-certificates\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:28.901758 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.901367    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4770276e-e9dc-4f6c-94f6-d2dfe2f6f115-tmp-dir\") pod \"node-resolver-49rdp\" (UID: \"4770276e-e9dc-4f6c-94f6-d2dfe2f6f115\") " pod="openshift-dns/node-resolver-49rdp"
Oct 30 00:41:28.901758 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.901379    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0142f4ce-76c5-4ca1-ac00-c2c334696905-serving-cert\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:28.901758 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.901478    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-certificates\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:28.913985 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.913964    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79mmb\" (UniqueName: \"kubernetes.io/projected/794aa402-53bf-43fc-b21b-a176bbbcd132-kube-api-access-79mmb\") pod \"cluster-samples-operator-6ddb6c8b94-gh5xc\" (UID: \"794aa402-53bf-43fc-b21b-a176bbbcd132\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"
Oct 30 00:41:28.914042 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.913992    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7glf\" (UniqueName: \"kubernetes.io/projected/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-kube-api-access-v7glf\") pod \"cluster-monitoring-operator-884576c57-lfnrl\" (UID: \"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:28.914116 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.914099    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpn92\" (UniqueName: \"kubernetes.io/projected/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-kube-api-access-bpn92\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:41:28.919487 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.919467    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmm4r\" (UniqueName: \"kubernetes.io/projected/59e5f12b-4a3e-449b-b285-17747780e1cf-kube-api-access-cmm4r\") pod \"service-ca-operator-544d749c8c-jbp7s\" (UID: \"59e5f12b-4a3e-449b-b285-17747780e1cf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s"
Oct 30 00:41:28.919700 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.919684    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s72cg\" (UniqueName: \"kubernetes.io/projected/a2dadf26-b255-4281-91f4-98536d370d21-kube-api-access-s72cg\") pod \"kas-connection-checker-685895c7c9-dtn28\" (UID: \"a2dadf26-b255-4281-91f4-98536d370d21\") " pod="kube-system/kas-connection-checker-685895c7c9-dtn28"
Oct 30 00:41:28.919823 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.919806    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxvpz\" (UniqueName: \"kubernetes.io/projected/0a50fdaf-7855-4033-96c8-6cb23212a986-kube-api-access-pxvpz\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:28.920123 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.920103    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8s6t\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-kube-api-access-d8s6t\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:28.920156 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.920110    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45zv7\" (UniqueName: \"kubernetes.io/projected/0142f4ce-76c5-4ca1-ac00-c2c334696905-kube-api-access-45zv7\") pod \"insights-operator-7c4986d449-xxvch\" (UID: \"0142f4ce-76c5-4ca1-ac00-c2c334696905\") " pod="openshift-insights/insights-operator-7c4986d449-xxvch"
Oct 30 00:41:28.922658 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.922641    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g77n6\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-kube-api-access-g77n6\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:28.924208 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.924190    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54t8p\" (UniqueName: \"kubernetes.io/projected/a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55-kube-api-access-54t8p\") pod \"ingress-canary-hf7tk\" (UID: \"a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55\") " pod="openshift-ingress-canary/ingress-canary-hf7tk"
Oct 30 00:41:28.924286 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.924268    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78557\" (UniqueName: \"kubernetes.io/projected/d95535e9-e677-463a-a662-7e8045c71d9b-kube-api-access-78557\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:28.924324 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.924311    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk7mm\" (UniqueName: \"kubernetes.io/projected/f5fade25-18f0-4e3f-978a-5bb39a356ab0-kube-api-access-tk7mm\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:41:28.924784 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.924769    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q94d\" (UniqueName: \"kubernetes.io/projected/433f04c0-a9ea-45c8-a2b7-245980607391-kube-api-access-4q94d\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs\" (UID: \"433f04c0-a9ea-45c8-a2b7-245980607391\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs"
Oct 30 00:41:28.924817 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.924784    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kksvb\" (UniqueName: \"kubernetes.io/projected/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-kube-api-access-kksvb\") pod \"network-metrics-daemon-b4686\" (UID: \"0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493\") " pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:41:28.924977 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.924962    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nkxz\" (UniqueName: \"kubernetes.io/projected/19a807d5-c007-47b4-953c-28de606ababc-kube-api-access-8nkxz\") pod \"node-ca-qdjmq\" (UID: \"19a807d5-c007-47b4-953c-28de606ababc\") " pod="openshift-image-registry/node-ca-qdjmq"
Oct 30 00:41:28.925082 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.925055    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tkrk\" (UniqueName: \"kubernetes.io/projected/4509811e-30a0-4c8b-b21e-58826bd1f6c6-kube-api-access-5tkrk\") pod \"kas-connection-checker-685895c7c9-6q8k5\" (UID: \"4509811e-30a0-4c8b-b21e-58826bd1f6c6\") " pod="kube-system/kas-connection-checker-685895c7c9-6q8k5"
Oct 30 00:41:28.925555 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.925540    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j89wv\" (UniqueName: \"kubernetes.io/projected/380df728-1bf4-4258-ba04-c31e1d28a3aa-kube-api-access-j89wv\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:28.931092 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.930845    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-bound-sa-token\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:28.931092 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.930863    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpx52\" (UniqueName: \"kubernetes.io/projected/fc3514a7-8b72-40ff-b45a-1c2005b59360-kube-api-access-cpx52\") pod \"kas-connection-checker-685895c7c9-pq5xl\" (UID: \"fc3514a7-8b72-40ff-b45a-1c2005b59360\") " pod="kube-system/kas-connection-checker-685895c7c9-pq5xl"
Oct 30 00:41:28.931092 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.930878    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75l4z\" (UniqueName: \"kubernetes.io/projected/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-kube-api-access-75l4z\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:28.931092 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.930978    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq6ls\" (UniqueName: \"kubernetes.io/projected/c75bc9c9-74be-412b-a5ac-794f971bea31-kube-api-access-jq6ls\") pod \"iptables-alerter-ljlrc\" (UID: \"c75bc9c9-74be-412b-a5ac-794f971bea31\") " pod="openshift-network-operator/iptables-alerter-ljlrc"
Oct 30 00:41:28.931351 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.931223    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vvj2\" (UniqueName: \"kubernetes.io/projected/204212cc-1c4c-4bfb-8b41-03a98042166c-kube-api-access-7vvj2\") pod \"network-check-target-jh728\" (UID: \"204212cc-1c4c-4bfb-8b41-03a98042166c\") " pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:41:28.931764 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.931697    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwnlf\" (UniqueName: \"kubernetes.io/projected/4770276e-e9dc-4f6c-94f6-d2dfe2f6f115-kube-api-access-lwnlf\") pod \"node-resolver-49rdp\" (UID: \"4770276e-e9dc-4f6c-94f6-d2dfe2f6f115\") " pod="openshift-dns/node-resolver-49rdp"
Oct 30 00:41:28.931853 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.931818    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sljzz\" (UniqueName: \"kubernetes.io/projected/8e4dbba6-28af-46b3-ac90-37ce437fd318-kube-api-access-sljzz\") pod \"console-operator-7cd56b8dbc-jk65g\" (UID: \"8e4dbba6-28af-46b3-ac90-37ce437fd318\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:28.932006 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.931881    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r64gt\" (UniqueName: \"kubernetes.io/projected/76b2bcb8-636c-46b5-8dfb-95831470349e-kube-api-access-r64gt\") pod \"network-check-source-6f78d59f99-9zmkk\" (UID: \"76b2bcb8-636c-46b5-8dfb-95831470349e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9zmkk"
Oct 30 00:41:28.932006 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.931997    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnf9q\" (UniqueName: \"kubernetes.io/projected/0eac4baa-d0d9-43c0-8f81-3733a026e091-kube-api-access-nnf9q\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:28.932490 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.932471    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8xgb\" (UniqueName: \"kubernetes.io/projected/e05b9b53-5f5b-402e-8599-ce61481346e6-kube-api-access-l8xgb\") pod \"volume-data-source-validator-58c9f5f46d-nkb8p\" (UID: \"e05b9b53-5f5b-402e-8599-ce61481346e6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nkb8p"
Oct 30 00:41:28.933097 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.933059    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-bound-sa-token\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:28.935592 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:28.935574    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpqz6\" (UniqueName: \"kubernetes.io/projected/ae0f3040-049e-4f4e-a6e5-206b31aeea55-kube-api-access-jpqz6\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:29.001880 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.001791    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lfnrl\" (UID: \"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:29.001880 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.001828    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-run-netns\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.001880 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.001848    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.001880 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.001864    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j7px8\" (UID: \"6a145873-a56a-4fe8-bdb5-dd0e6837b948\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j7px8"
Oct 30 00:41:29.002231 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.001918    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-run-netns\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.002231 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.001948    4864 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 00:41:29.002231 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.001951    4864 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 00:41:29.002231 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.001986    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-host\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:29.002231 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.001918    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.002231 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.002014    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert podName:6a145873-a56a-4fe8-bdb5-dd0e6837b948 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:29.501998238 +0000 UTC m=+1.840100126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j7px8" (UID: "6a145873-a56a-4fe8-bdb5-dd0e6837b948") : secret "networking-console-plugin-cert" not found
Oct 30 00:41:29.002231 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.002042    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls podName:2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc nodeName:}" failed. No retries permitted until 2026-10-30 00:41:29.502023672 +0000 UTC m=+1.840125568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lfnrl" (UID: "2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc") : secret "cluster-monitoring-operator-tls" not found
Oct 30 00:41:29.002231 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002046    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-host\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:29.002231 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002108    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:29.002231 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002140    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-modprobe-d\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:29.002231 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002173    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-lib-modules\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:29.002231 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002199    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-device-dir\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:41:29.002231 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.002226    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle podName:d95535e9-e677-463a-a662-7e8045c71d9b nodeName:}" failed. No retries permitted until 2026-10-30 00:41:29.502214446 +0000 UTC m=+1.840316347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle") pod "router-default-94c9fcc47-fdtjt" (UID: "d95535e9-e677-463a-a662-7e8045c71d9b") : configmap references non-existent config key: service-ca.crt
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002243    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-multus-cni-dir\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002261    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-device-dir\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002268    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-sys\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002294    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-cni-bin\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002306    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-multus-cni-dir\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002269    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-modprobe-d\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002320    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-sys-fs\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002330    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-lib-modules\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002344    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-run-multus-certs\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002319    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-sys\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002351    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-cni-bin\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002369    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002387    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-sys-fs\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002377    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-run-multus-certs\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002409    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5fade25-18f0-4e3f-978a-5bb39a356ab0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.002416    4864 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.002435    4864 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-654dd5d568-jdc79: secret "image-registry-tls" not found
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002453    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5fade25-18f0-4e3f-978a-5bb39a356ab0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002479    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-run-ovn-kubernetes\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.002501    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls podName:c48ee8c6-60b2-4229-a013-4199c44813e4 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:29.502490312 +0000 UTC m=+1.840592205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls") pod "image-registry-654dd5d568-jdc79" (UID: "c48ee8c6-60b2-4229-a013-4199c44813e4") : secret "image-registry-tls" not found
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002517    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-run-ovn-kubernetes\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002518    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-etc-kubernetes\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002540    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbef968a-7aa4-4e23-9278-3706e60d8c0b-dbus\") pod \"global-pull-secret-syncer-wk65n\" (UID: \"bbef968a-7aa4-4e23-9278-3706e60d8c0b\") " pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002555    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-systemd-units\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002570    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gh5xc\" (UID: \"794aa402-53bf-43fc-b21b-a176bbbcd132\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002576    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-etc-kubernetes\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002595    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-multus-socket-dir-parent\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002621    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-systemd-units\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002639    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-run-openvswitch\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002644    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbef968a-7aa4-4e23-9278-3706e60d8c0b-dbus\") pod \"global-pull-secret-syncer-wk65n\" (UID: \"bbef968a-7aa4-4e23-9278-3706e60d8c0b\") " pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002674    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-multus-socket-dir-parent\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002680    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-run-openvswitch\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002694    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a50fdaf-7855-4033-96c8-6cb23212a986-crio-socket\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002716    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbef968a-7aa4-4e23-9278-3706e60d8c0b-kubelet-config\") pod \"global-pull-secret-syncer-wk65n\" (UID: \"bbef968a-7aa4-4e23-9278-3706e60d8c0b\") " pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002755    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbef968a-7aa4-4e23-9278-3706e60d8c0b-kubelet-config\") pod \"global-pull-secret-syncer-wk65n\" (UID: \"bbef968a-7aa4-4e23-9278-3706e60d8c0b\") " pod="kube-system/global-pull-secret-syncer-wk65n"
Oct 30 00:41:29.002743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002758    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a50fdaf-7855-4033-96c8-6cb23212a986-crio-socket\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002783    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-kubelet\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002798    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-var-lib-openvswitch\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002844    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a50fdaf-7855-4033-96c8-6cb23212a986-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002849    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-kubelet\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002862    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-var-lib-cni-multus\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002883    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c75bc9c9-74be-412b-a5ac-794f971bea31-host-slash\") pod \"iptables-alerter-ljlrc\" (UID: \"c75bc9c9-74be-412b-a5ac-794f971bea31\") " pod="openshift-network-operator/iptables-alerter-ljlrc"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002898    4864 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002911    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-var-lib-openvswitch\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002915    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-var-lib-cni-multus\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002907    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002949    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c75bc9c9-74be-412b-a5ac-794f971bea31-host-slash\") pod \"iptables-alerter-ljlrc\" (UID: \"c75bc9c9-74be-412b-a5ac-794f971bea31\") " pod="openshift-network-operator/iptables-alerter-ljlrc"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.002967    4864 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.002976    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-system-cni-dir\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.002998    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls podName:cee8ca5c-ba97-47b5-b22f-2e161bb93fdc nodeName:}" failed. No retries permitted until 2026-10-30 00:41:29.502988498 +0000 UTC m=+1.841090386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls") pod "dns-default-wq8t9" (UID: "cee8ca5c-ba97-47b5-b22f-2e161bb93fdc") : secret "dns-default-metrics-tls" not found
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003013    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-systemd\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003032    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-system-cni-dir\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003033    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-run\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003062    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003064    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-run\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003102    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-var-lib-cni-bin\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003103    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-systemd\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003118    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-multus-conf-dir\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003141    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-multus-conf-dir\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003146    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-socket-dir\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.003171    4864 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003172    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-etc-openvswitch\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.003182    4864 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f45784fb5-r5j44: secret "image-registry-tls" not found
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003196    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-sysctl-conf\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003218    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5fade25-18f0-4e3f-978a-5bb39a356ab0-cnibin\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003216    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-socket-dir\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.003230    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls podName:564f9ba3-d19e-4cac-9021-81939eb8e631 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:29.503217359 +0000 UTC m=+1.841319264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls") pod "image-registry-5f45784fb5-r5j44" (UID: "564f9ba3-d19e-4cac-9021-81939eb8e631") : secret "image-registry-tls" not found
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003250    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5fade25-18f0-4e3f-978a-5bb39a356ab0-cnibin\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003261    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-sysctl-d\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003285    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-etc-openvswitch\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003251    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-var-lib-cni-bin\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003326    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-log-socket\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003295    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-log-socket\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003358    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-sysctl-d\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003364    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003387    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-cnibin\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003411    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-cni-netd\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003418    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-sysctl-conf\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003434    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-sysconfig\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003466    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-kubernetes\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003472    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-cni-netd\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003477    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a50fdaf-7855-4033-96c8-6cb23212a986-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003494    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003502    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-sysconfig\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003512    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003532    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-run-netns\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.003548    4864 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003558    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-etc-kubernetes\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003559    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-slash\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003473    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-cnibin\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.003584    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs podName:d95535e9-e677-463a-a662-7e8045c71d9b nodeName:}" failed. No retries permitted until 2026-10-30 00:41:29.50357427 +0000 UTC m=+1.841676160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs") pod "router-default-94c9fcc47-fdtjt" (UID: "d95535e9-e677-463a-a662-7e8045c71d9b") : secret "router-metrics-certs-default" not found
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003575    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-run-netns\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003593    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-host-slash\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003615    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-run-ovn\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003633    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5fade25-18f0-4e3f-978a-5bb39a356ab0-system-cni-dir\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003648    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-etc-selinux\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003667    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-run-systemd\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003684    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-run-ovn\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003693    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-var-lib-kubelet\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003704    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5fade25-18f0-4e3f-978a-5bb39a356ab0-system-cni-dir\") pod \"multus-additional-cni-plugins-tz5vn\" (UID: \"f5fade25-18f0-4e3f-978a-5bb39a356ab0\") " pod="openshift-multus/multus-additional-cni-plugins-tz5vn"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003733    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-etc-selinux\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003744    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-run-systemd\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003737    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-var-lib-kubelet\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003784    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-var-lib-kubelet\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003814    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19a807d5-c007-47b4-953c-28de606ababc-host\") pod \"node-ca-qdjmq\" (UID: \"19a807d5-c007-47b4-953c-28de606ababc\") " pod="openshift-image-registry/node-ca-qdjmq"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003840    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-run-k8s-cni-cncf-io\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003864    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-hostroot\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003899    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19a807d5-c007-47b4-953c-28de606ababc-host\") pod \"node-ca-qdjmq\" (UID: \"19a807d5-c007-47b4-953c-28de606ababc\") " pod="openshift-image-registry/node-ca-qdjmq"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003902    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-registration-dir\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003904    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-host-run-k8s-cni-cncf-io\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003904    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0eac4baa-d0d9-43c0-8f81-3733a026e091-hostroot\") pod \"multus-qp98l\" (UID: \"0eac4baa-d0d9-43c0-8f81-3733a026e091\") " pod="openshift-multus/multus-qp98l"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003925    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-node-log\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003873    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae0f3040-049e-4f4e-a6e5-206b31aeea55-var-lib-kubelet\") pod \"tuned-q4bch\" (UID: \"ae0f3040-049e-4f4e-a6e5-206b31aeea55\") " pod="openshift-cluster-node-tuning-operator/tuned-q4bch"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003937    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074-registration-dir\") pod \"aws-ebs-csi-driver-node-9cfzn\" (UID: \"e7d77e73-a34c-4df8-b1fc-d8e3c1d2e074\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cfzn"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003949    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs\") pod \"network-metrics-daemon-b4686\" (UID: \"0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493\") " pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003975    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4770276e-e9dc-4f6c-94f6-d2dfe2f6f115-hosts-file\") pod \"node-resolver-49rdp\" (UID: \"4770276e-e9dc-4f6c-94f6-d2dfe2f6f115\") " pod="openshift-dns/node-resolver-49rdp"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.003976    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/380df728-1bf4-4258-ba04-c31e1d28a3aa-node-log\") pod \"ovnkube-node-7855k\" (UID: \"380df728-1bf4-4258-ba04-c31e1d28a3aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.004020    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4770276e-e9dc-4f6c-94f6-d2dfe2f6f115-hosts-file\") pod \"node-resolver-49rdp\" (UID: \"4770276e-e9dc-4f6c-94f6-d2dfe2f6f115\") " pod="openshift-dns/node-resolver-49rdp"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.004023    4864 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.004112    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs podName:0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:29.504096678 +0000 UTC m=+1.842198583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs") pod "network-metrics-daemon-b4686" (UID: "0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493") : secret "metrics-daemon-secret" not found
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.004035    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.004164    4864 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:29.004230 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.004205    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls podName:0a50fdaf-7855-4033-96c8-6cb23212a986 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:29.504191117 +0000 UTC m=+1.842293018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f9vrm" (UID: "0a50fdaf-7855-4033-96c8-6cb23212a986") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:29.007795 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.005320    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/794aa402-53bf-43fc-b21b-a176bbbcd132-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gh5xc\" (UID: \"794aa402-53bf-43fc-b21b-a176bbbcd132\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"
Oct 30 00:41:29.085527 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.085490    4864 scope.go:122] "RemoveContainer" containerID="a31b42b1c0dd5bc05a1170b6f3ab972f96c72f863fca50fdd3997fb3c7fd610b"
Oct 30 00:41:29.089963 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.089944    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"
Oct 30 00:41:29.433227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.433194    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc"]
Oct 30 00:41:29.489617 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.489601    4864 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:41:29.509200 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.509175    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lfnrl\" (UID: \"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:29.509323 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.509208    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j7px8\" (UID: \"6a145873-a56a-4fe8-bdb5-dd0e6837b948\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j7px8"
Oct 30 00:41:29.509323 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.509231    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:29.509323 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.509259    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:29.509323 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.509304    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:29.509323 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.509317    4864 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 00:41:29.509539 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.509330    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:29.509539 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.509318    4864 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 00:41:29.509539 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.509367    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle podName:d95535e9-e677-463a-a662-7e8045c71d9b nodeName:}" failed. No retries permitted until 2026-10-30 00:41:30.50934893 +0000 UTC m=+2.847450831 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle") pod "router-default-94c9fcc47-fdtjt" (UID: "d95535e9-e677-463a-a662-7e8045c71d9b") : configmap references non-existent config key: service-ca.crt
Oct 30 00:41:29.509539 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.509369    4864 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:41:29.509539 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.509392    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert podName:6a145873-a56a-4fe8-bdb5-dd0e6837b948 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:30.509383143 +0000 UTC m=+2.847485033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j7px8" (UID: "6a145873-a56a-4fe8-bdb5-dd0e6837b948") : secret "networking-console-plugin-cert" not found
Oct 30 00:41:29.509539 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.509394    4864 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-654dd5d568-jdc79: secret "image-registry-tls" not found
Oct 30 00:41:29.509539 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.509408    4864 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:41:29.509539 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.509413    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls podName:2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc nodeName:}" failed. No retries permitted until 2026-10-30 00:41:30.509402512 +0000 UTC m=+2.847504415 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lfnrl" (UID: "2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc") : secret "cluster-monitoring-operator-tls" not found
Oct 30 00:41:29.509539 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.509421    4864 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f45784fb5-r5j44: secret "image-registry-tls" not found
Oct 30 00:41:29.509539 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.509421    4864 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:41:29.509539 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.509463    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls podName:564f9ba3-d19e-4cac-9021-81939eb8e631 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:30.509453624 +0000 UTC m=+2.847555530 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls") pod "image-registry-5f45784fb5-r5j44" (UID: "564f9ba3-d19e-4cac-9021-81939eb8e631") : secret "image-registry-tls" not found
Oct 30 00:41:29.509539 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.509479    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:29.509539 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.509510    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs\") pod \"network-metrics-daemon-b4686\" (UID: \"0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493\") " pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:41:29.509539 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.509538    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls podName:cee8ca5c-ba97-47b5-b22f-2e161bb93fdc nodeName:}" failed. No retries permitted until 2026-10-30 00:41:30.509525138 +0000 UTC m=+2.847627033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls") pod "dns-default-wq8t9" (UID: "cee8ca5c-ba97-47b5-b22f-2e161bb93fdc") : secret "dns-default-metrics-tls" not found
Oct 30 00:41:29.510016 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.509555    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls podName:c48ee8c6-60b2-4229-a013-4199c44813e4 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:30.509547893 +0000 UTC m=+2.847649788 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls") pod "image-registry-654dd5d568-jdc79" (UID: "c48ee8c6-60b2-4229-a013-4199c44813e4") : secret "image-registry-tls" not found
Oct 30 00:41:29.510016 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.509576    4864 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 00:41:29.510016 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.509575    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:29.510016 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.509607    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs podName:0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:30.509598214 +0000 UTC m=+2.847700107 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs") pod "network-metrics-daemon-b4686" (UID: "0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493") : secret "metrics-daemon-secret" not found
Oct 30 00:41:29.510016 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.509637    4864 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 00:41:29.510016 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.509653    4864 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:29.510016 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.509674    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs podName:d95535e9-e677-463a-a662-7e8045c71d9b nodeName:}" failed. No retries permitted until 2026-10-30 00:41:30.509661923 +0000 UTC m=+2.847763830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs") pod "router-default-94c9fcc47-fdtjt" (UID: "d95535e9-e677-463a-a662-7e8045c71d9b") : secret "router-metrics-certs-default" not found
Oct 30 00:41:29.510016 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.509689    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls podName:0a50fdaf-7855-4033-96c8-6cb23212a986 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:30.509683255 +0000 UTC m=+2.847785143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f9vrm" (UID: "0a50fdaf-7855-4033-96c8-6cb23212a986") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:29.605592 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.605523    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-49rdp_4770276e-e9dc-4f6c-94f6-d2dfe2f6f115/dns-node-resolver/0.log"
Oct 30 00:41:29.843448 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.843420    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 00:41:29.843829 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.843780    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/1.log"
Oct 30 00:41:29.843829 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.843812    4864 generic.go:356] "Generic (PLEG): container finished" podID="8e4dbba6-28af-46b3-ac90-37ce437fd318" containerID="581b2515fd8461d8e3abcc9ccef91c19f42bea200061d8bf44fb90057b38c86e" exitCode=255
Oct 30 00:41:29.843904 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.843882    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g" event={"ID":"8e4dbba6-28af-46b3-ac90-37ce437fd318","Type":"ContainerDied","Data":"581b2515fd8461d8e3abcc9ccef91c19f42bea200061d8bf44fb90057b38c86e"}
Oct 30 00:41:29.843948 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.843937    4864 scope.go:122] "RemoveContainer" containerID="a31b42b1c0dd5bc05a1170b6f3ab972f96c72f863fca50fdd3997fb3c7fd610b"
Oct 30 00:41:29.844314 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.844290    4864 scope.go:122] "RemoveContainer" containerID="581b2515fd8461d8e3abcc9ccef91c19f42bea200061d8bf44fb90057b38c86e"
Oct 30 00:41:29.844510 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:29.844492    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jk65g_openshift-console-operator(8e4dbba6-28af-46b3-ac90-37ce437fd318)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g" podUID="8e4dbba6-28af-46b3-ac90-37ce437fd318"
Oct 30 00:41:29.845169 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:29.845144    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc" event={"ID":"794aa402-53bf-43fc-b21b-a176bbbcd132","Type":"ContainerStarted","Data":"25c1b34d99748e9da56f23e8fa053ece87c70096818b6b8e415222207d543397"}
Oct 30 00:41:30.369903 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:30.369871    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:41:30.518621 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:30.518583    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs\") pod \"network-metrics-daemon-b4686\" (UID: \"0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493\") " pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:41:30.518621 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:30.518621    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:30.518855 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:30.518645    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lfnrl\" (UID: \"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:30.518855 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:30.518675    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j7px8\" (UID: \"6a145873-a56a-4fe8-bdb5-dd0e6837b948\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j7px8"
Oct 30 00:41:30.518855 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:30.518731    4864 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:30.518855 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:30.518742    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:30.518855 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:30.518783    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:30.518855 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:30.518795    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls podName:0a50fdaf-7855-4033-96c8-6cb23212a986 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:32.518776697 +0000 UTC m=+4.856878600 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f9vrm" (UID: "0a50fdaf-7855-4033-96c8-6cb23212a986") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:30.518855 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:30.518795    4864 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 00:41:30.518855 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:30.518730    4864 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 00:41:30.518855 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:30.518845    4864 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:41:30.518855 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:30.518803    4864 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 00:41:30.518855 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:30.518851    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:30.518855 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:30.518859    4864 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-654dd5d568-jdc79: secret "image-registry-tls" not found
Oct 30 00:41:30.519245 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:30.518860    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls podName:2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc nodeName:}" failed. No retries permitted until 2026-10-30 00:41:32.518843232 +0000 UTC m=+4.856945137 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lfnrl" (UID: "2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc") : secret "cluster-monitoring-operator-tls" not found
Oct 30 00:41:30.519245 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:30.518909    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs podName:0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:32.518898223 +0000 UTC m=+4.857000112 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs") pod "network-metrics-daemon-b4686" (UID: "0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493") : secret "metrics-daemon-secret" not found
Oct 30 00:41:30.519245 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:30.518925    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle podName:d95535e9-e677-463a-a662-7e8045c71d9b nodeName:}" failed. No retries permitted until 2026-10-30 00:41:32.518917049 +0000 UTC m=+4.857018937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle") pod "router-default-94c9fcc47-fdtjt" (UID: "d95535e9-e677-463a-a662-7e8045c71d9b") : configmap references non-existent config key: service-ca.crt
Oct 30 00:41:30.519245 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:30.518951    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert podName:6a145873-a56a-4fe8-bdb5-dd0e6837b948 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:32.5189371 +0000 UTC m=+4.857038997 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j7px8" (UID: "6a145873-a56a-4fe8-bdb5-dd0e6837b948") : secret "networking-console-plugin-cert" not found
Oct 30 00:41:30.519245 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:30.518970    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls podName:c48ee8c6-60b2-4229-a013-4199c44813e4 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:32.5189633 +0000 UTC m=+4.857065188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls") pod "image-registry-654dd5d568-jdc79" (UID: "c48ee8c6-60b2-4229-a013-4199c44813e4") : secret "image-registry-tls" not found
Oct 30 00:41:30.519245 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:30.518993    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:30.519245 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:30.519057    4864 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:41:30.519245 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:30.519084    4864 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f45784fb5-r5j44: secret "image-registry-tls" not found
Oct 30 00:41:30.519245 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:30.519087    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:30.519245 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:30.519120    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls podName:564f9ba3-d19e-4cac-9021-81939eb8e631 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:32.519109269 +0000 UTC m=+4.857211157 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls") pod "image-registry-5f45784fb5-r5j44" (UID: "564f9ba3-d19e-4cac-9021-81939eb8e631") : secret "image-registry-tls" not found
Oct 30 00:41:30.521191 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:30.521176    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cee8ca5c-ba97-47b5-b22f-2e161bb93fdc-metrics-tls\") pod \"dns-default-wq8t9\" (UID: \"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc\") " pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:30.521293 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:30.521273    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95535e9-e677-463a-a662-7e8045c71d9b-metrics-certs\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:30.597515 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:30.597482    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:30.851376 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:30.851337    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 00:41:30.919980 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:30.919899    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wq8t9"]
Oct 30 00:41:30.926227 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:30.926186    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee8ca5c_ba97_47b5_b22f_2e161bb93fdc.slice/crio-3143b391d282ed2e968fd7848c83b3a0c2be43e783e83dab6258dce9512accf7 WatchSource:0}: Error finding container 3143b391d282ed2e968fd7848c83b3a0c2be43e783e83dab6258dce9512accf7: Status 404 returned error can't find the container with id 3143b391d282ed2e968fd7848c83b3a0c2be43e783e83dab6258dce9512accf7
Oct 30 00:41:31.003891 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:31.003855    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qdjmq_19a807d5-c007-47b4-953c-28de606ababc/node-ca/0.log"
Oct 30 00:41:31.803931 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:31.803905    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hf7tk_a2b0e334-b89a-4ef4-91ac-dc0ca93a8b55/serve-healthcheck-canary/0.log"
Oct 30 00:41:31.857647 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:31.857608    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc" event={"ID":"794aa402-53bf-43fc-b21b-a176bbbcd132","Type":"ContainerStarted","Data":"ce26a736b92313d22237a2c240846855c949174636ae255e71afecaafbda0d7c"}
Oct 30 00:41:31.858063 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:31.857676    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gh5xc" event={"ID":"794aa402-53bf-43fc-b21b-a176bbbcd132","Type":"ContainerStarted","Data":"9d44c313404cef4441b4882d874144e2d380deb13a848a2384727751851457ec"}
Oct 30 00:41:31.858882 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:31.858858    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wq8t9" event={"ID":"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc","Type":"ContainerStarted","Data":"3143b391d282ed2e968fd7848c83b3a0c2be43e783e83dab6258dce9512accf7"}
Oct 30 00:41:32.373702 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:32.373669    4864 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77dcb_2ac86565-2bfb-45e4-840d-c0f3a29cdfa5/konnectivity-agent/0.log" limit=1024
Oct 30 00:41:32.404025 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:32.404002    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs_433f04c0-a9ea-45c8-a2b7-245980607391/kube-storage-version-migrator-operator/0.log"
Oct 30 00:41:32.538117 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:32.538012    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs\") pod \"network-metrics-daemon-b4686\" (UID: \"0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493\") " pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:41:32.538117 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:32.538052    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:32.538117 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:32.538095    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lfnrl\" (UID: \"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:32.538435 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:32.538128    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j7px8\" (UID: \"6a145873-a56a-4fe8-bdb5-dd0e6837b948\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j7px8"
Oct 30 00:41:32.538435 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:32.538168    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:32.538435 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:32.538203    4864 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:32.538435 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:32.538265    4864 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:41:32.538435 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:32.538204    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:32.538435 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:32.538265    4864 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 00:41:32.538435 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:32.538271    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls podName:0a50fdaf-7855-4033-96c8-6cb23212a986 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:36.5382532 +0000 UTC m=+8.876355107 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f9vrm" (UID: "0a50fdaf-7855-4033-96c8-6cb23212a986") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:32.538435 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:32.538277    4864 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-654dd5d568-jdc79: secret "image-registry-tls" not found
Oct 30 00:41:32.538435 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:32.538357    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle podName:d95535e9-e677-463a-a662-7e8045c71d9b nodeName:}" failed. No retries permitted until 2026-10-30 00:41:36.538334342 +0000 UTC m=+8.876436246 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle") pod "router-default-94c9fcc47-fdtjt" (UID: "d95535e9-e677-463a-a662-7e8045c71d9b") : configmap references non-existent config key: service-ca.crt
Oct 30 00:41:32.538435 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:32.538265    4864 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 00:41:32.538435 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:32.538373    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert podName:6a145873-a56a-4fe8-bdb5-dd0e6837b948 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:36.53836682 +0000 UTC m=+8.876468708 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j7px8" (UID: "6a145873-a56a-4fe8-bdb5-dd0e6837b948") : secret "networking-console-plugin-cert" not found
Oct 30 00:41:32.538435 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:32.538387    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls podName:c48ee8c6-60b2-4229-a013-4199c44813e4 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:36.53838023 +0000 UTC m=+8.876482118 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls") pod "image-registry-654dd5d568-jdc79" (UID: "c48ee8c6-60b2-4229-a013-4199c44813e4") : secret "image-registry-tls" not found
Oct 30 00:41:32.538435 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:32.538399    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls podName:2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc nodeName:}" failed. No retries permitted until 2026-10-30 00:41:36.538393962 +0000 UTC m=+8.876495850 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lfnrl" (UID: "2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc") : secret "cluster-monitoring-operator-tls" not found
Oct 30 00:41:32.538879 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:32.538456    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:32.538879 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:32.538542    4864 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:41:32.538879 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:32.538550    4864 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f45784fb5-r5j44: secret "image-registry-tls" not found
Oct 30 00:41:32.538879 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:32.538579    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls podName:564f9ba3-d19e-4cac-9021-81939eb8e631 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:36.538573168 +0000 UTC m=+8.876675055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls") pod "image-registry-5f45784fb5-r5j44" (UID: "564f9ba3-d19e-4cac-9021-81939eb8e631") : secret "image-registry-tls" not found
Oct 30 00:41:32.540462 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:32.540438    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493-metrics-certs\") pod \"network-metrics-daemon-b4686\" (UID: \"0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493\") " pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:41:32.622768 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:32.622739    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:32.693258 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:32.693225    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b4686"
Oct 30 00:41:32.864419 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:32.864326    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wq8t9" event={"ID":"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc","Type":"ContainerStarted","Data":"10ae4eeb868afffe00992d4d67d303abf7b17f0a0c90ce82b699db34f1aa1ac5"}
Oct 30 00:41:32.864419 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:32.864389    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wq8t9" event={"ID":"cee8ca5c-ba97-47b5-b22f-2e161bb93fdc","Type":"ContainerStarted","Data":"9382fecac75deed8787386cb0d4ceffe0f09b981cdad74404ec4582266319dbe"}
Oct 30 00:41:32.864945 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:32.864678    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:32.984618 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:32.984585    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b4686"]
Oct 30 00:41:32.990281 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:32.990256    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9a4b99_85aa_4f0b_9ea4_4f928b9b1493.slice/crio-1fa640e5c38d0eb2f38a68eb5fa39ac0bc821d81b99eb54abb0bfbd8b4331b85 WatchSource:0}: Error finding container 1fa640e5c38d0eb2f38a68eb5fa39ac0bc821d81b99eb54abb0bfbd8b4331b85: Status 404 returned error can't find the container with id 1fa640e5c38d0eb2f38a68eb5fa39ac0bc821d81b99eb54abb0bfbd8b4331b85
Oct 30 00:41:33.872186 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:33.872150    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b4686" event={"ID":"0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493","Type":"ContainerStarted","Data":"33d37748c202a44b726ef9da5d0ef210b6366ce571d7f6d0e12b5661f757df1e"}
Oct 30 00:41:33.872484 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:33.872219    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b4686" event={"ID":"0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493","Type":"ContainerStarted","Data":"1fa640e5c38d0eb2f38a68eb5fa39ac0bc821d81b99eb54abb0bfbd8b4331b85"}
Oct 30 00:41:34.076400 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:34.076316    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:34.076703 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:34.076685    4864 scope.go:122] "RemoveContainer" containerID="581b2515fd8461d8e3abcc9ccef91c19f42bea200061d8bf44fb90057b38c86e"
Oct 30 00:41:34.076870 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:34.076852    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jk65g_openshift-console-operator(8e4dbba6-28af-46b3-ac90-37ce437fd318)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g" podUID="8e4dbba6-28af-46b3-ac90-37ce437fd318"
Oct 30 00:41:34.382884 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:34.382814    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:34.383017 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:34.383007    4864 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:41:34.877529 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:34.877492    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b4686" event={"ID":"0c9a4b99-85aa-4f0b-9ea4-4f928b9b1493","Type":"ContainerStarted","Data":"92afa091bb80ab6813cbaa4e75c3a23f929bf93d5ea917486bdc1c589ef72b8d"}
Oct 30 00:41:36.578246 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:36.578207    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:36.578246 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:36.578249    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lfnrl\" (UID: \"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:36.578783 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:36.578271    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j7px8\" (UID: \"6a145873-a56a-4fe8-bdb5-dd0e6837b948\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j7px8"
Oct 30 00:41:36.578783 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:36.578291    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:36.578783 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:36.578316    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:36.578783 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:36.578347    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:36.578783 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:36.578379    4864 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:36.578783 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:36.578445    4864 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:41:36.578783 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:36.578459    4864 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f45784fb5-r5j44: secret "image-registry-tls" not found
Oct 30 00:41:36.578783 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:36.578466    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls podName:0a50fdaf-7855-4033-96c8-6cb23212a986 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:44.578446271 +0000 UTC m=+16.916548178 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f9vrm" (UID: "0a50fdaf-7855-4033-96c8-6cb23212a986") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:36.578783 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:36.578483    4864 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 00:41:36.578783 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:36.578499    4864 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-654dd5d568-jdc79: secret "image-registry-tls" not found
Oct 30 00:41:36.578783 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:36.578504    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls podName:564f9ba3-d19e-4cac-9021-81939eb8e631 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:44.578489503 +0000 UTC m=+16.916591405 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls") pod "image-registry-5f45784fb5-r5j44" (UID: "564f9ba3-d19e-4cac-9021-81939eb8e631") : secret "image-registry-tls" not found
Oct 30 00:41:36.578783 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:36.578550    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls podName:c48ee8c6-60b2-4229-a013-4199c44813e4 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:44.578537121 +0000 UTC m=+16.916639021 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls") pod "image-registry-654dd5d568-jdc79" (UID: "c48ee8c6-60b2-4229-a013-4199c44813e4") : secret "image-registry-tls" not found
Oct 30 00:41:36.579232 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:36.578967    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95535e9-e677-463a-a662-7e8045c71d9b-service-ca-bundle\") pod \"router-default-94c9fcc47-fdtjt\" (UID: \"d95535e9-e677-463a-a662-7e8045c71d9b\") " pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:36.580813 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:36.580792    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a145873-a56a-4fe8-bdb5-dd0e6837b948-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j7px8\" (UID: \"6a145873-a56a-4fe8-bdb5-dd0e6837b948\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j7px8"
Oct 30 00:41:36.580851 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:36.580792    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lfnrl\" (UID: \"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:36.586543 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:36.586523    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"
Oct 30 00:41:36.590022 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:36.589991    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:36.590155 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:36.590140    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-j7px8"
Oct 30 00:41:36.811885 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:36.811839    4864 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:36.812226 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:36.812212    4864 scope.go:122] "RemoveContainer" containerID="581b2515fd8461d8e3abcc9ccef91c19f42bea200061d8bf44fb90057b38c86e"
Oct 30 00:41:36.812403 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:36.812386    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jk65g_openshift-console-operator(8e4dbba6-28af-46b3-ac90-37ce437fd318)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g" podUID="8e4dbba6-28af-46b3-ac90-37ce437fd318"
Oct 30 00:41:36.893107 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:36.893019    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-j7px8"]
Oct 30 00:41:36.899408 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:36.899380    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a145873_a56a_4fe8_bdb5_dd0e6837b948.slice/crio-dc61d303cbc3493e86b69e8e1a05a33cdd66e0fd5f2bb90294e2209ac3762160 WatchSource:0}: Error finding container dc61d303cbc3493e86b69e8e1a05a33cdd66e0fd5f2bb90294e2209ac3762160: Status 404 returned error can't find the container with id dc61d303cbc3493e86b69e8e1a05a33cdd66e0fd5f2bb90294e2209ac3762160
Oct 30 00:41:36.906489 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:36.906459    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl"]
Oct 30 00:41:36.911628 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:36.911608    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f315dfa_46fe_400e_aef8_8f5f2cbe2ccc.slice/crio-fbb88d039d423728742204b17799472e22d943b707d4d4794a28f8fa4d6d6032 WatchSource:0}: Error finding container fbb88d039d423728742204b17799472e22d943b707d4d4794a28f8fa4d6d6032: Status 404 returned error can't find the container with id fbb88d039d423728742204b17799472e22d943b707d4d4794a28f8fa4d6d6032
Oct 30 00:41:36.921922 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:36.921897    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-94c9fcc47-fdtjt"]
Oct 30 00:41:36.926686 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:36.926658    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95535e9_e677_463a_a662_7e8045c71d9b.slice/crio-a5f8147965595f54cdbdcc5f53c5422487cedc0be7163e1fa1b400cc238b1f20 WatchSource:0}: Error finding container a5f8147965595f54cdbdcc5f53c5422487cedc0be7163e1fa1b400cc238b1f20: Status 404 returned error can't find the container with id a5f8147965595f54cdbdcc5f53c5422487cedc0be7163e1fa1b400cc238b1f20
Oct 30 00:41:36.930228 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:36.930191    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-77dcb"
Oct 30 00:41:37.889237 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:37.889187    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl" event={"ID":"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc","Type":"ContainerStarted","Data":"fbb88d039d423728742204b17799472e22d943b707d4d4794a28f8fa4d6d6032"}
Oct 30 00:41:37.890461 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:37.890433    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-j7px8" event={"ID":"6a145873-a56a-4fe8-bdb5-dd0e6837b948","Type":"ContainerStarted","Data":"dc61d303cbc3493e86b69e8e1a05a33cdd66e0fd5f2bb90294e2209ac3762160"}
Oct 30 00:41:37.891932 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:37.891896    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-94c9fcc47-fdtjt" event={"ID":"d95535e9-e677-463a-a662-7e8045c71d9b","Type":"ContainerStarted","Data":"e6f369e605fb727b60d046070dd59a9d6afb82f691b6800126ba3a6513b1bde2"}
Oct 30 00:41:37.892012 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:37.891941    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-94c9fcc47-fdtjt" event={"ID":"d95535e9-e677-463a-a662-7e8045c71d9b","Type":"ContainerStarted","Data":"a5f8147965595f54cdbdcc5f53c5422487cedc0be7163e1fa1b400cc238b1f20"}
Oct 30 00:41:38.049764 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:38.049732    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7855k"
Oct 30 00:41:38.590332 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:38.590296    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:38.592718 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:38.592696    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:38.900132 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:38.900089    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-j7px8" event={"ID":"6a145873-a56a-4fe8-bdb5-dd0e6837b948","Type":"ContainerStarted","Data":"9336bddb06876400220675e5719db133054b226f0f4b1585f7403c2ea0e990ee"}
Oct 30 00:41:38.905044 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:38.905018    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lfnrl" event={"ID":"2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc","Type":"ContainerStarted","Data":"e6c203310b27ceab23bb5b93e83122b95becf62e2e4568fee308e66004f32d42"}
Oct 30 00:41:39.478312 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:39.478275    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:39.479568 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:39.479544    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-94c9fcc47-fdtjt"
Oct 30 00:41:40.786133 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:40.786103    4864 scope.go:122] "RemoveContainer" containerID="581b2515fd8461d8e3abcc9ccef91c19f42bea200061d8bf44fb90057b38c86e"
Oct 30 00:41:40.912561 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:40.912532    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 00:41:40.912749 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:40.912589    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g" event={"ID":"8e4dbba6-28af-46b3-ac90-37ce437fd318","Type":"ContainerStarted","Data":"66bf373e7ffea01f2a3d38974cf160cf5ecf867d5ad3b8babfd864a25840a4de"}
Oct 30 00:41:40.913047 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:40.913025    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:40.913880 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:40.913854    4864 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-jk65g container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.11:8443/readyz\": dial tcp 10.132.0.11:8443: connect: connection refused" start-of-body=
Oct 30 00:41:40.913963 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:40.913908    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g" podUID="8e4dbba6-28af-46b3-ac90-37ce437fd318" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.11:8443/readyz\": dial tcp 10.132.0.11:8443: connect: connection refused"
Oct 30 00:41:41.920202 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:41.920176    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk65g"
Oct 30 00:41:42.874889 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:42.874859    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wq8t9"
Oct 30 00:41:44.651349 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:44.651314    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:44.651835 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:44.651361    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:44.651835 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:44.651390    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:44.653804 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:44.653776    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls\") pod \"image-registry-5f45784fb5-r5j44\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") " pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:44.653923 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:44.653802    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a50fdaf-7855-4033-96c8-6cb23212a986-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f9vrm\" (UID: \"0a50fdaf-7855-4033-96c8-6cb23212a986\") " pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:44.653923 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:44.653805    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls\") pod \"image-registry-654dd5d568-jdc79\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") " pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:44.689835 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:44.689808    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:44.694461 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:44.694444    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:44.697465 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:44.697414    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f9vrm"
Oct 30 00:41:45.012646 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:45.012612    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f9vrm"]
Oct 30 00:41:45.017887 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:45.017861    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a50fdaf_7855_4033_96c8_6cb23212a986.slice/crio-847c96fb306e65c162ea9d6e6d06a4e1ce5038b200e27d56030d1fe7d16e1425 WatchSource:0}: Error finding container 847c96fb306e65c162ea9d6e6d06a4e1ce5038b200e27d56030d1fe7d16e1425: Status 404 returned error can't find the container with id 847c96fb306e65c162ea9d6e6d06a4e1ce5038b200e27d56030d1fe7d16e1425
Oct 30 00:41:45.020895 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:45.020870    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod564f9ba3_d19e_4cac_9021_81939eb8e631.slice/crio-8a0c32f093903366013cc2448295c21fda273da9df35c09e13fe812176aa4b46 WatchSource:0}: Error finding container 8a0c32f093903366013cc2448295c21fda273da9df35c09e13fe812176aa4b46: Status 404 returned error can't find the container with id 8a0c32f093903366013cc2448295c21fda273da9df35c09e13fe812176aa4b46
Oct 30 00:41:45.023494 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:45.023455    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-654dd5d568-jdc79"]
Oct 30 00:41:45.026046 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:45.026027    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f45784fb5-r5j44"]
Oct 30 00:41:45.027011 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:45.026986    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc48ee8c6_60b2_4229_a013_4199c44813e4.slice/crio-3b45ba0e8dd192ea7376a6b088699f9b967e3d34e1494eca7998c766230212ce WatchSource:0}: Error finding container 3b45ba0e8dd192ea7376a6b088699f9b967e3d34e1494eca7998c766230212ce: Status 404 returned error can't find the container with id 3b45ba0e8dd192ea7376a6b088699f9b967e3d34e1494eca7998c766230212ce
Oct 30 00:41:45.930571 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:45.930474    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f45784fb5-r5j44" event={"ID":"564f9ba3-d19e-4cac-9021-81939eb8e631","Type":"ContainerStarted","Data":"f202d72f81e3554e635b1528f18bf7a8b3e6afa98cdcf3ef69899627c867ed92"}
Oct 30 00:41:45.930571 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:45.930534    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f45784fb5-r5j44" event={"ID":"564f9ba3-d19e-4cac-9021-81939eb8e631","Type":"ContainerStarted","Data":"8a0c32f093903366013cc2448295c21fda273da9df35c09e13fe812176aa4b46"}
Oct 30 00:41:45.931061 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:45.930613    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:41:45.932224 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:45.932200    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-654dd5d568-jdc79" event={"ID":"c48ee8c6-60b2-4229-a013-4199c44813e4","Type":"ContainerStarted","Data":"ec57208086c1008bfab99a1cd4aee6e976c24b1e0c3855de9377019254451fbf"}
Oct 30 00:41:45.932360 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:45.932239    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-654dd5d568-jdc79" event={"ID":"c48ee8c6-60b2-4229-a013-4199c44813e4","Type":"ContainerStarted","Data":"3b45ba0e8dd192ea7376a6b088699f9b967e3d34e1494eca7998c766230212ce"}
Oct 30 00:41:45.932360 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:45.932263    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:41:45.933935 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:45.933917    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f9vrm" event={"ID":"0a50fdaf-7855-4033-96c8-6cb23212a986","Type":"ContainerStarted","Data":"e58338c005840108cf8f653669fb8d8664e83fcb360c2282c55c3d4bdb29136f"}
Oct 30 00:41:45.934038 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:45.933962    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f9vrm" event={"ID":"0a50fdaf-7855-4033-96c8-6cb23212a986","Type":"ContainerStarted","Data":"20439a749488ac45afd9346e6032bf89fbab1879df92d86c25fcdfb9133335c7"}
Oct 30 00:41:45.934038 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:45.933992    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f9vrm" event={"ID":"0a50fdaf-7855-4033-96c8-6cb23212a986","Type":"ContainerStarted","Data":"847c96fb306e65c162ea9d6e6d06a4e1ce5038b200e27d56030d1fe7d16e1425"}
Oct 30 00:41:46.439146 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:46.439061    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-654dd5d568-jdc79"]
Oct 30 00:41:46.939113 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:46.939054    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f9vrm" event={"ID":"0a50fdaf-7855-4033-96c8-6cb23212a986","Type":"ContainerStarted","Data":"6ce120cbb9d4a074257a081e4b8f3ec7e472ba6baa1c42c36fbb111ab1f5ce8b"}
Oct 30 00:41:49.037536 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:41:49.037491    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:41:54.388846 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.388812    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mhjdw"]
Oct 30 00:41:54.432204 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.432177    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.499326 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.499280    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 00:41:54.499995 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.499962    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6r6tq\""
Oct 30 00:41:54.499995 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.499367    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 00:41:54.500202 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.500032    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 00:41:54.500299 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.500284    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 00:41:54.526620 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.526591    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8661055c-1d07-4776-96f7-ca8e36c1ea84-node-exporter-textfile\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.526755 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.526626    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8661055c-1d07-4776-96f7-ca8e36c1ea84-metrics-client-ca\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.526755 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.526652    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhv74\" (UniqueName: \"kubernetes.io/projected/8661055c-1d07-4776-96f7-ca8e36c1ea84-kube-api-access-vhv74\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.526875 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.526762    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8661055c-1d07-4776-96f7-ca8e36c1ea84-sys\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.526875 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.526784    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8661055c-1d07-4776-96f7-ca8e36c1ea84-root\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.526875 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.526813    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8661055c-1d07-4776-96f7-ca8e36c1ea84-node-exporter-tls\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.527029 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.526912    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8661055c-1d07-4776-96f7-ca8e36c1ea84-node-exporter-wtmp\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.527029 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.526944    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8661055c-1d07-4776-96f7-ca8e36c1ea84-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.527168 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.527045    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8661055c-1d07-4776-96f7-ca8e36c1ea84-node-exporter-accelerators-collector-config\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.628250 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.628213    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8661055c-1d07-4776-96f7-ca8e36c1ea84-node-exporter-wtmp\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.628403 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.628312    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8661055c-1d07-4776-96f7-ca8e36c1ea84-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.628403 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.628363    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8661055c-1d07-4776-96f7-ca8e36c1ea84-node-exporter-accelerators-collector-config\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.628403 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.628359    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8661055c-1d07-4776-96f7-ca8e36c1ea84-node-exporter-wtmp\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.628503 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.628464    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8661055c-1d07-4776-96f7-ca8e36c1ea84-node-exporter-textfile\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.628503 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.628490    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8661055c-1d07-4776-96f7-ca8e36c1ea84-metrics-client-ca\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.628571 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.628504    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhv74\" (UniqueName: \"kubernetes.io/projected/8661055c-1d07-4776-96f7-ca8e36c1ea84-kube-api-access-vhv74\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.628571 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.628540    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8661055c-1d07-4776-96f7-ca8e36c1ea84-sys\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.628667 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.628573    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8661055c-1d07-4776-96f7-ca8e36c1ea84-root\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.628667 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.628642    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8661055c-1d07-4776-96f7-ca8e36c1ea84-node-exporter-tls\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.628772 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.628682    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8661055c-1d07-4776-96f7-ca8e36c1ea84-sys\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.628772 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.628731    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8661055c-1d07-4776-96f7-ca8e36c1ea84-root\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.628898 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.628876    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8661055c-1d07-4776-96f7-ca8e36c1ea84-node-exporter-textfile\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.628973 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.628950    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8661055c-1d07-4776-96f7-ca8e36c1ea84-node-exporter-accelerators-collector-config\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.629136 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.629120    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8661055c-1d07-4776-96f7-ca8e36c1ea84-metrics-client-ca\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.630672 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.630649    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8661055c-1d07-4776-96f7-ca8e36c1ea84-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.630911 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.630895    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8661055c-1d07-4776-96f7-ca8e36c1ea84-node-exporter-tls\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.649537 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.649481    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhv74\" (UniqueName: \"kubernetes.io/projected/8661055c-1d07-4776-96f7-ca8e36c1ea84-kube-api-access-vhv74\") pod \"node-exporter-mhjdw\" (UID: \"8661055c-1d07-4776-96f7-ca8e36c1ea84\") " pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.797862 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.797829    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mhjdw"
Oct 30 00:41:54.829811 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:41:54.829780    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8661055c_1d07_4776_96f7_ca8e36c1ea84.slice/crio-82a1e93cf201aede4f00c334c66e46a2906de9131336f8bbcffbf626310dd75b WatchSource:0}: Error finding container 82a1e93cf201aede4f00c334c66e46a2906de9131336f8bbcffbf626310dd75b: Status 404 returned error can't find the container with id 82a1e93cf201aede4f00c334c66e46a2906de9131336f8bbcffbf626310dd75b
Oct 30 00:41:54.969031 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:54.968998    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mhjdw" event={"ID":"8661055c-1d07-4776-96f7-ca8e36c1ea84","Type":"ContainerStarted","Data":"82a1e93cf201aede4f00c334c66e46a2906de9131336f8bbcffbf626310dd75b"}
Oct 30 00:41:56.977693 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:56.977662    4864 generic.go:356] "Generic (PLEG): container finished" podID="8661055c-1d07-4776-96f7-ca8e36c1ea84" containerID="3e9fdf854928fa814db70d5322e26f35c4ee2a1a0a1e46359fff9919a96bf843" exitCode=0
Oct 30 00:41:56.978059 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:56.977699    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mhjdw" event={"ID":"8661055c-1d07-4776-96f7-ca8e36c1ea84","Type":"ContainerDied","Data":"3e9fdf854928fa814db70d5322e26f35c4ee2a1a0a1e46359fff9919a96bf843"}
Oct 30 00:41:57.983542 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:57.983506    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mhjdw" event={"ID":"8661055c-1d07-4776-96f7-ca8e36c1ea84","Type":"ContainerStarted","Data":"cf1a125b2f0123927c78b2513de916b683f8c78b75b37b9e44e2f874f0dacfd4"}
Oct 30 00:41:57.984030 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:57.983553    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mhjdw" event={"ID":"8661055c-1d07-4776-96f7-ca8e36c1ea84","Type":"ContainerStarted","Data":"5fbc8c09fda035eba535c72fbd8dd5cb0f04c6a40e14ada3c338aaf503f2d95e"}
Oct 30 00:41:58.043028 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:41:58.042973    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mhjdw" podStartSLOduration=2.576155531 podStartE2EDuration="4.042960027s" totalImagesPullingTime="1.399093952s" totalInitContainerRuntime="67.710544ms" isStatefulPod=true podCreationTimestamp="2026-10-30 00:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:58.032127278 +0000 UTC m=+30.370229192" watchObservedRunningTime="2026-10-30 00:41:58.042960027 +0000 UTC m=+30.381061999"
Oct 30 00:42:00.372542 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:00.372507    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jh728"
Oct 30 00:42:06.943520 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:06.943492    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:42:07.948192 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:07.948167    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:42:09.325225 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:09.325192    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f45784fb5-r5j44"]
Oct 30 00:42:12.967270 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:12.967150    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-654dd5d568-jdc79" podUID="c48ee8c6-60b2-4229-a013-4199c44813e4" containerName="registry" containerID="cri-o://ec57208086c1008bfab99a1cd4aee6e976c24b1e0c3855de9377019254451fbf" gracePeriod=30
Oct 30 00:42:13.212367 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.212341    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:42:13.273732 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.273644    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c48ee8c6-60b2-4229-a013-4199c44813e4-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c48ee8c6-60b2-4229-a013-4199c44813e4-image-registry-private-configuration\") pod \"c48ee8c6-60b2-4229-a013-4199c44813e4\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") "
Oct 30 00:42:13.273732 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.273698    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-bound-sa-token\") pod \"c48ee8c6-60b2-4229-a013-4199c44813e4\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") "
Oct 30 00:42:13.273973 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.273816    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c48ee8c6-60b2-4229-a013-4199c44813e4-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c48ee8c6-60b2-4229-a013-4199c44813e4-installation-pull-secrets\") pod \"c48ee8c6-60b2-4229-a013-4199c44813e4\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") "
Oct 30 00:42:13.273973 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.273865    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c48ee8c6-60b2-4229-a013-4199c44813e4-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c48ee8c6-60b2-4229-a013-4199c44813e4-trusted-ca\") pod \"c48ee8c6-60b2-4229-a013-4199c44813e4\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") "
Oct 30 00:42:13.273973 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.273909    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls\") pod \"c48ee8c6-60b2-4229-a013-4199c44813e4\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") "
Oct 30 00:42:13.273973 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.273962    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-certificates\") pod \"c48ee8c6-60b2-4229-a013-4199c44813e4\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") "
Oct 30 00:42:13.274217 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.273996    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-kube-api-access-g77n6\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-kube-api-access-g77n6\") pod \"c48ee8c6-60b2-4229-a013-4199c44813e4\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") "
Oct 30 00:42:13.274217 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.274034    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c48ee8c6-60b2-4229-a013-4199c44813e4-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c48ee8c6-60b2-4229-a013-4199c44813e4-ca-trust-extracted\") pod \"c48ee8c6-60b2-4229-a013-4199c44813e4\" (UID: \"c48ee8c6-60b2-4229-a013-4199c44813e4\") "
Oct 30 00:42:13.274758 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.274633    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c48ee8c6-60b2-4229-a013-4199c44813e4-trusted-ca" pod "c48ee8c6-60b2-4229-a013-4199c44813e4" (UID: "c48ee8c6-60b2-4229-a013-4199c44813e4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:42:13.274758 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.274711    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-certificates" pod "c48ee8c6-60b2-4229-a013-4199c44813e4" (UID: "c48ee8c6-60b2-4229-a013-4199c44813e4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:42:13.276821 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.276765    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-bound-sa-token" pod "c48ee8c6-60b2-4229-a013-4199c44813e4" (UID: "c48ee8c6-60b2-4229-a013-4199c44813e4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:42:13.277143 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.277110    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls" pod "c48ee8c6-60b2-4229-a013-4199c44813e4" (UID: "c48ee8c6-60b2-4229-a013-4199c44813e4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:42:13.277240 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.277155    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-kube-api-access-g77n6" pod "c48ee8c6-60b2-4229-a013-4199c44813e4" (UID: "c48ee8c6-60b2-4229-a013-4199c44813e4"). InnerVolumeSpecName "kube-api-access-g77n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:42:13.277380 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.277355    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c48ee8c6-60b2-4229-a013-4199c44813e4-installation-pull-secrets" pod "c48ee8c6-60b2-4229-a013-4199c44813e4" (UID: "c48ee8c6-60b2-4229-a013-4199c44813e4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:42:13.283283 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.283255    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c48ee8c6-60b2-4229-a013-4199c44813e4-ca-trust-extracted" pod "c48ee8c6-60b2-4229-a013-4199c44813e4" (UID: "c48ee8c6-60b2-4229-a013-4199c44813e4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:42:13.283805 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.283734    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c48ee8c6-60b2-4229-a013-4199c44813e4-image-registry-private-configuration" pod "c48ee8c6-60b2-4229-a013-4199c44813e4" (UID: "c48ee8c6-60b2-4229-a013-4199c44813e4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:42:13.374794 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.374758    4864 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c48ee8c6-60b2-4229-a013-4199c44813e4-image-registry-private-configuration\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:42:13.374794 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.374789    4864 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-bound-sa-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:42:13.374794 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.374801    4864 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c48ee8c6-60b2-4229-a013-4199c44813e4-installation-pull-secrets\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:42:13.375002 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.374812    4864 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c48ee8c6-60b2-4229-a013-4199c44813e4-trusted-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:42:13.375002 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.374822    4864 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:42:13.375002 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.374831    4864 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c48ee8c6-60b2-4229-a013-4199c44813e4-registry-certificates\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:42:13.375002 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.374840    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g77n6\" (UniqueName: \"kubernetes.io/projected/c48ee8c6-60b2-4229-a013-4199c44813e4-kube-api-access-g77n6\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:42:13.375002 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:13.374848    4864 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c48ee8c6-60b2-4229-a013-4199c44813e4-ca-trust-extracted\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:42:14.051578 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:14.051494    4864 generic.go:356] "Generic (PLEG): container finished" podID="c48ee8c6-60b2-4229-a013-4199c44813e4" containerID="ec57208086c1008bfab99a1cd4aee6e976c24b1e0c3855de9377019254451fbf" exitCode=0
Oct 30 00:42:14.051578 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:14.051540    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-654dd5d568-jdc79"
Oct 30 00:42:14.051952 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:14.051577    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-654dd5d568-jdc79" event={"ID":"c48ee8c6-60b2-4229-a013-4199c44813e4","Type":"ContainerDied","Data":"ec57208086c1008bfab99a1cd4aee6e976c24b1e0c3855de9377019254451fbf"}
Oct 30 00:42:14.051952 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:14.051634    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-654dd5d568-jdc79" event={"ID":"c48ee8c6-60b2-4229-a013-4199c44813e4","Type":"ContainerDied","Data":"3b45ba0e8dd192ea7376a6b088699f9b967e3d34e1494eca7998c766230212ce"}
Oct 30 00:42:14.051952 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:14.051657    4864 scope.go:122] "RemoveContainer" containerID="ec57208086c1008bfab99a1cd4aee6e976c24b1e0c3855de9377019254451fbf"
Oct 30 00:42:14.060495 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:14.060477    4864 scope.go:122] "RemoveContainer" containerID="ec57208086c1008bfab99a1cd4aee6e976c24b1e0c3855de9377019254451fbf"
Oct 30 00:42:14.060788 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:42:14.060759    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec57208086c1008bfab99a1cd4aee6e976c24b1e0c3855de9377019254451fbf\": container with ID starting with ec57208086c1008bfab99a1cd4aee6e976c24b1e0c3855de9377019254451fbf not found: ID does not exist" containerID="ec57208086c1008bfab99a1cd4aee6e976c24b1e0c3855de9377019254451fbf"
Oct 30 00:42:14.060828 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:14.060796    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec57208086c1008bfab99a1cd4aee6e976c24b1e0c3855de9377019254451fbf"} err="failed to get container status \"ec57208086c1008bfab99a1cd4aee6e976c24b1e0c3855de9377019254451fbf\": rpc error: code = NotFound desc = could not find container \"ec57208086c1008bfab99a1cd4aee6e976c24b1e0c3855de9377019254451fbf\": container with ID starting with ec57208086c1008bfab99a1cd4aee6e976c24b1e0c3855de9377019254451fbf not found: ID does not exist"
Oct 30 00:42:14.107292 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:14.107263    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-654dd5d568-jdc79"]
Oct 30 00:42:14.117013 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:14.116991    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-654dd5d568-jdc79"]
Oct 30 00:42:15.790853 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:15.790820    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c48ee8c6-60b2-4229-a013-4199c44813e4" path="/var/lib/kubelet/pods/c48ee8c6-60b2-4229-a013-4199c44813e4/volumes"
Oct 30 00:42:19.037701 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:42:19.037664    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:42:34.344568 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.344521    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5f45784fb5-r5j44" podUID="564f9ba3-d19e-4cac-9021-81939eb8e631" containerName="registry" containerID="cri-o://f202d72f81e3554e635b1528f18bf7a8b3e6afa98cdcf3ef69899627c867ed92" gracePeriod=30
Oct 30 00:42:34.589272 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.589245    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:42:34.752794 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.752738    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/564f9ba3-d19e-4cac-9021-81939eb8e631-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/564f9ba3-d19e-4cac-9021-81939eb8e631-ca-trust-extracted\") pod \"564f9ba3-d19e-4cac-9021-81939eb8e631\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") "
Oct 30 00:42:34.752986 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.752833    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-certificates\") pod \"564f9ba3-d19e-4cac-9021-81939eb8e631\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") "
Oct 30 00:42:34.753057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.752984    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-bound-sa-token\") pod \"564f9ba3-d19e-4cac-9021-81939eb8e631\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") "
Oct 30 00:42:34.753057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.753017    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-kube-api-access-d8s6t\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-kube-api-access-d8s6t\") pod \"564f9ba3-d19e-4cac-9021-81939eb8e631\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") "
Oct 30 00:42:34.753057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.753042    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/564f9ba3-d19e-4cac-9021-81939eb8e631-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/564f9ba3-d19e-4cac-9021-81939eb8e631-installation-pull-secrets\") pod \"564f9ba3-d19e-4cac-9021-81939eb8e631\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") "
Oct 30 00:42:34.753257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.753060    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls\") pod \"564f9ba3-d19e-4cac-9021-81939eb8e631\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") "
Oct 30 00:42:34.753257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.753109    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/564f9ba3-d19e-4cac-9021-81939eb8e631-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/564f9ba3-d19e-4cac-9021-81939eb8e631-image-registry-private-configuration\") pod \"564f9ba3-d19e-4cac-9021-81939eb8e631\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") "
Oct 30 00:42:34.753257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.753149    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/564f9ba3-d19e-4cac-9021-81939eb8e631-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/564f9ba3-d19e-4cac-9021-81939eb8e631-trusted-ca\") pod \"564f9ba3-d19e-4cac-9021-81939eb8e631\" (UID: \"564f9ba3-d19e-4cac-9021-81939eb8e631\") "
Oct 30 00:42:34.753736 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.753701    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-certificates" pod "564f9ba3-d19e-4cac-9021-81939eb8e631" (UID: "564f9ba3-d19e-4cac-9021-81939eb8e631"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:42:34.753836 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.753747    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/564f9ba3-d19e-4cac-9021-81939eb8e631-trusted-ca" pod "564f9ba3-d19e-4cac-9021-81939eb8e631" (UID: "564f9ba3-d19e-4cac-9021-81939eb8e631"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:42:34.755640 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.755588    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls" pod "564f9ba3-d19e-4cac-9021-81939eb8e631" (UID: "564f9ba3-d19e-4cac-9021-81939eb8e631"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:42:34.755810 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.755781    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-kube-api-access-d8s6t" pod "564f9ba3-d19e-4cac-9021-81939eb8e631" (UID: "564f9ba3-d19e-4cac-9021-81939eb8e631"). InnerVolumeSpecName "kube-api-access-d8s6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:42:34.755933 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.755802    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/564f9ba3-d19e-4cac-9021-81939eb8e631-installation-pull-secrets" pod "564f9ba3-d19e-4cac-9021-81939eb8e631" (UID: "564f9ba3-d19e-4cac-9021-81939eb8e631"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:42:34.755933 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.755838    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-bound-sa-token" pod "564f9ba3-d19e-4cac-9021-81939eb8e631" (UID: "564f9ba3-d19e-4cac-9021-81939eb8e631"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:42:34.756148 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.756124    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/564f9ba3-d19e-4cac-9021-81939eb8e631-image-registry-private-configuration" pod "564f9ba3-d19e-4cac-9021-81939eb8e631" (UID: "564f9ba3-d19e-4cac-9021-81939eb8e631"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:42:34.759921 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.759839    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564f9ba3-d19e-4cac-9021-81939eb8e631-ca-trust-extracted" pod "564f9ba3-d19e-4cac-9021-81939eb8e631" (UID: "564f9ba3-d19e-4cac-9021-81939eb8e631"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:42:34.854068 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.854018    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8s6t\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-kube-api-access-d8s6t\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:42:34.854068 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.854063    4864 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/564f9ba3-d19e-4cac-9021-81939eb8e631-installation-pull-secrets\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:42:34.854068 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.854090    4864 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:42:34.854068 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.854101    4864 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/564f9ba3-d19e-4cac-9021-81939eb8e631-image-registry-private-configuration\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:42:34.854399 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.854111    4864 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/564f9ba3-d19e-4cac-9021-81939eb8e631-trusted-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:42:34.854399 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.854120    4864 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/564f9ba3-d19e-4cac-9021-81939eb8e631-ca-trust-extracted\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:42:34.854399 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.854128    4864 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/564f9ba3-d19e-4cac-9021-81939eb8e631-registry-certificates\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:42:34.854399 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:34.854137    4864 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/564f9ba3-d19e-4cac-9021-81939eb8e631-bound-sa-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:42:35.129764 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:35.129675    4864 generic.go:356] "Generic (PLEG): container finished" podID="564f9ba3-d19e-4cac-9021-81939eb8e631" containerID="f202d72f81e3554e635b1528f18bf7a8b3e6afa98cdcf3ef69899627c867ed92" exitCode=0
Oct 30 00:42:35.129764 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:35.129723    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f45784fb5-r5j44"
Oct 30 00:42:35.129978 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:35.129768    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f45784fb5-r5j44" event={"ID":"564f9ba3-d19e-4cac-9021-81939eb8e631","Type":"ContainerDied","Data":"f202d72f81e3554e635b1528f18bf7a8b3e6afa98cdcf3ef69899627c867ed92"}
Oct 30 00:42:35.129978 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:35.129819    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f45784fb5-r5j44" event={"ID":"564f9ba3-d19e-4cac-9021-81939eb8e631","Type":"ContainerDied","Data":"8a0c32f093903366013cc2448295c21fda273da9df35c09e13fe812176aa4b46"}
Oct 30 00:42:35.129978 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:35.129839    4864 scope.go:122] "RemoveContainer" containerID="f202d72f81e3554e635b1528f18bf7a8b3e6afa98cdcf3ef69899627c867ed92"
Oct 30 00:42:35.138137 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:35.138118    4864 scope.go:122] "RemoveContainer" containerID="f202d72f81e3554e635b1528f18bf7a8b3e6afa98cdcf3ef69899627c867ed92"
Oct 30 00:42:35.138423 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:42:35.138398    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f202d72f81e3554e635b1528f18bf7a8b3e6afa98cdcf3ef69899627c867ed92\": container with ID starting with f202d72f81e3554e635b1528f18bf7a8b3e6afa98cdcf3ef69899627c867ed92 not found: ID does not exist" containerID="f202d72f81e3554e635b1528f18bf7a8b3e6afa98cdcf3ef69899627c867ed92"
Oct 30 00:42:35.138486 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:35.138436    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f202d72f81e3554e635b1528f18bf7a8b3e6afa98cdcf3ef69899627c867ed92"} err="failed to get container status \"f202d72f81e3554e635b1528f18bf7a8b3e6afa98cdcf3ef69899627c867ed92\": rpc error: code = NotFound desc = could not find container \"f202d72f81e3554e635b1528f18bf7a8b3e6afa98cdcf3ef69899627c867ed92\": container with ID starting with f202d72f81e3554e635b1528f18bf7a8b3e6afa98cdcf3ef69899627c867ed92 not found: ID does not exist"
Oct 30 00:42:35.185434 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:35.185401    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f45784fb5-r5j44"]
Oct 30 00:42:35.195666 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:35.195643    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5f45784fb5-r5j44"]
Oct 30 00:42:35.790209 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:35.790177    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="564f9ba3-d19e-4cac-9021-81939eb8e631" path="/var/lib/kubelet/pods/564f9ba3-d19e-4cac-9021-81939eb8e631/volumes"
Oct 30 00:42:38.145571 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:38.145532    4864 generic.go:356] "Generic (PLEG): container finished" podID="59e5f12b-4a3e-449b-b285-17747780e1cf" containerID="7bca7e28adb91f11f321ad0f34a0d5c247b3030455427436d7a921d2a064af19" exitCode=0
Oct 30 00:42:38.145970 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:38.145608    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s" event={"ID":"59e5f12b-4a3e-449b-b285-17747780e1cf","Type":"ContainerDied","Data":"7bca7e28adb91f11f321ad0f34a0d5c247b3030455427436d7a921d2a064af19"}
Oct 30 00:42:38.145970 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:38.145906    4864 scope.go:122] "RemoveContainer" containerID="7bca7e28adb91f11f321ad0f34a0d5c247b3030455427436d7a921d2a064af19"
Oct 30 00:42:39.150615 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:39.150582    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jbp7s" event={"ID":"59e5f12b-4a3e-449b-b285-17747780e1cf","Type":"ContainerStarted","Data":"a7bf3a7d63bbd258e86cc0cc208b87b1f457e77d3d517d73857cbc050b6aba27"}
Oct 30 00:42:48.187027 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:48.186948    4864 generic.go:356] "Generic (PLEG): container finished" podID="433f04c0-a9ea-45c8-a2b7-245980607391" containerID="3b8903639645aeb40bd7c4b0c07088c77293e2bd82c74f2c9159990d372cd7ee" exitCode=0
Oct 30 00:42:48.187027 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:48.186997    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs" event={"ID":"433f04c0-a9ea-45c8-a2b7-245980607391","Type":"ContainerDied","Data":"3b8903639645aeb40bd7c4b0c07088c77293e2bd82c74f2c9159990d372cd7ee"}
Oct 30 00:42:48.187423 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:48.187291    4864 scope.go:122] "RemoveContainer" containerID="3b8903639645aeb40bd7c4b0c07088c77293e2bd82c74f2c9159990d372cd7ee"
Oct 30 00:42:49.039184 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:42:49.039144    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:42:49.192190 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:49.192154    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gpxgs" event={"ID":"433f04c0-a9ea-45c8-a2b7-245980607391","Type":"ContainerStarted","Data":"06a2da10d1eff498046bb05aa0b7177fd3564e8c71b425a13050ba796f63ffc7"}
Oct 30 00:42:53.207630 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:53.207592    4864 generic.go:356] "Generic (PLEG): container finished" podID="0142f4ce-76c5-4ca1-ac00-c2c334696905" containerID="9e812370545aa77a80f3b5432fcab16e59428b16387c8758193886d0053838fd" exitCode=0
Oct 30 00:42:53.208010 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:53.207662    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xxvch" event={"ID":"0142f4ce-76c5-4ca1-ac00-c2c334696905","Type":"ContainerDied","Data":"9e812370545aa77a80f3b5432fcab16e59428b16387c8758193886d0053838fd"}
Oct 30 00:42:53.208010 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:53.207949    4864 scope.go:122] "RemoveContainer" containerID="9e812370545aa77a80f3b5432fcab16e59428b16387c8758193886d0053838fd"
Oct 30 00:42:53.974751 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:53.974713    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-ljbgk-w44n4"]
Oct 30 00:42:53.975093 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:53.975058    4864 cpu_manager.go:414] "containerMap: removing container" podUID="564f9ba3-d19e-4cac-9021-81939eb8e631" containerName="registry"
Oct 30 00:42:53.975240 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:53.975092    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="564f9ba3-d19e-4cac-9021-81939eb8e631" containerName="registry"
Oct 30 00:42:53.975240 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:53.975110    4864 cpu_manager.go:414] "containerMap: removing container" podUID="c48ee8c6-60b2-4229-a013-4199c44813e4" containerName="registry"
Oct 30 00:42:53.975240 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:53.975116    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c48ee8c6-60b2-4229-a013-4199c44813e4" containerName="registry"
Oct 30 00:42:53.975240 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:53.975205    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="564f9ba3-d19e-4cac-9021-81939eb8e631" containerName="registry"
Oct 30 00:42:53.975240 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:53.975216    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="c48ee8c6-60b2-4229-a013-4199c44813e4" containerName="registry"
Oct 30 00:42:53.978296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:53.978276    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ljbgk-w44n4"
Oct 30 00:42:53.991171 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:53.991153    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-748hz\""
Oct 30 00:42:54.002378 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:54.002354    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-ljbgk-w44n4"]
Oct 30 00:42:54.109931 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:54.109902    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxns4\" (UniqueName: \"kubernetes.io/projected/829d9860-08f5-4ccf-8ba6-d53907fc44b6-kube-api-access-jxns4\") pod \"periodic-gathering-ljbgk-w44n4\" (UID: \"829d9860-08f5-4ccf-8ba6-d53907fc44b6\") " pod="openshift-insights/periodic-gathering-ljbgk-w44n4"
Oct 30 00:42:54.110129 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:54.109949    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/829d9860-08f5-4ccf-8ba6-d53907fc44b6-archives-path\") pod \"periodic-gathering-ljbgk-w44n4\" (UID: \"829d9860-08f5-4ccf-8ba6-d53907fc44b6\") " pod="openshift-insights/periodic-gathering-ljbgk-w44n4"
Oct 30 00:42:54.110129 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:54.110023    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/829d9860-08f5-4ccf-8ba6-d53907fc44b6-service-ca-bundle\") pod \"periodic-gathering-ljbgk-w44n4\" (UID: \"829d9860-08f5-4ccf-8ba6-d53907fc44b6\") " pod="openshift-insights/periodic-gathering-ljbgk-w44n4"
Oct 30 00:42:54.110240 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:54.110128    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/829d9860-08f5-4ccf-8ba6-d53907fc44b6-trusted-ca-bundle\") pod \"periodic-gathering-ljbgk-w44n4\" (UID: \"829d9860-08f5-4ccf-8ba6-d53907fc44b6\") " pod="openshift-insights/periodic-gathering-ljbgk-w44n4"
Oct 30 00:42:54.211082 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:54.211038    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxns4\" (UniqueName: \"kubernetes.io/projected/829d9860-08f5-4ccf-8ba6-d53907fc44b6-kube-api-access-jxns4\") pod \"periodic-gathering-ljbgk-w44n4\" (UID: \"829d9860-08f5-4ccf-8ba6-d53907fc44b6\") " pod="openshift-insights/periodic-gathering-ljbgk-w44n4"
Oct 30 00:42:54.211517 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:54.211120    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/829d9860-08f5-4ccf-8ba6-d53907fc44b6-archives-path\") pod \"periodic-gathering-ljbgk-w44n4\" (UID: \"829d9860-08f5-4ccf-8ba6-d53907fc44b6\") " pod="openshift-insights/periodic-gathering-ljbgk-w44n4"
Oct 30 00:42:54.211517 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:54.211153    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/829d9860-08f5-4ccf-8ba6-d53907fc44b6-service-ca-bundle\") pod \"periodic-gathering-ljbgk-w44n4\" (UID: \"829d9860-08f5-4ccf-8ba6-d53907fc44b6\") " pod="openshift-insights/periodic-gathering-ljbgk-w44n4"
Oct 30 00:42:54.211517 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:54.211186    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/829d9860-08f5-4ccf-8ba6-d53907fc44b6-trusted-ca-bundle\") pod \"periodic-gathering-ljbgk-w44n4\" (UID: \"829d9860-08f5-4ccf-8ba6-d53907fc44b6\") " pod="openshift-insights/periodic-gathering-ljbgk-w44n4"
Oct 30 00:42:54.211642 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:54.211573    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/829d9860-08f5-4ccf-8ba6-d53907fc44b6-archives-path\") pod \"periodic-gathering-ljbgk-w44n4\" (UID: \"829d9860-08f5-4ccf-8ba6-d53907fc44b6\") " pod="openshift-insights/periodic-gathering-ljbgk-w44n4"
Oct 30 00:42:54.211907 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:54.211884    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/829d9860-08f5-4ccf-8ba6-d53907fc44b6-service-ca-bundle\") pod \"periodic-gathering-ljbgk-w44n4\" (UID: \"829d9860-08f5-4ccf-8ba6-d53907fc44b6\") " pod="openshift-insights/periodic-gathering-ljbgk-w44n4"
Oct 30 00:42:54.212004 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:54.211985    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/829d9860-08f5-4ccf-8ba6-d53907fc44b6-trusted-ca-bundle\") pod \"periodic-gathering-ljbgk-w44n4\" (UID: \"829d9860-08f5-4ccf-8ba6-d53907fc44b6\") " pod="openshift-insights/periodic-gathering-ljbgk-w44n4"
Oct 30 00:42:54.212945 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:54.212919    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xxvch" event={"ID":"0142f4ce-76c5-4ca1-ac00-c2c334696905","Type":"ContainerStarted","Data":"7096c2edf6fec8749ce3141f36c0186590b6c542d10c15fc2b8768291e51c415"}
Oct 30 00:42:54.232639 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:54.232582    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxns4\" (UniqueName: \"kubernetes.io/projected/829d9860-08f5-4ccf-8ba6-d53907fc44b6-kube-api-access-jxns4\") pod \"periodic-gathering-ljbgk-w44n4\" (UID: \"829d9860-08f5-4ccf-8ba6-d53907fc44b6\") " pod="openshift-insights/periodic-gathering-ljbgk-w44n4"
Oct 30 00:42:54.289703 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:54.289677    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ljbgk-w44n4"
Oct 30 00:42:54.584160 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:54.584085    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-ljbgk-w44n4"]
Oct 30 00:42:54.589397 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:42:54.589366    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod829d9860_08f5_4ccf_8ba6_d53907fc44b6.slice/crio-0960c9eb6fa6cb85719d00b3add82872bae08e1547e2d333deb6d0362bd4157d WatchSource:0}: Error finding container 0960c9eb6fa6cb85719d00b3add82872bae08e1547e2d333deb6d0362bd4157d: Status 404 returned error can't find the container with id 0960c9eb6fa6cb85719d00b3add82872bae08e1547e2d333deb6d0362bd4157d
Oct 30 00:42:55.218205 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:55.218169    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ljbgk-w44n4" event={"ID":"829d9860-08f5-4ccf-8ba6-d53907fc44b6","Type":"ContainerStarted","Data":"d88730b8e6560680fc47c67c989d4ab056e3a6a5fbebc8c7aa7a4c4fb41fdff9"}
Oct 30 00:42:55.218602 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:55.218222    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ljbgk-w44n4" event={"ID":"829d9860-08f5-4ccf-8ba6-d53907fc44b6","Type":"ContainerStarted","Data":"0960c9eb6fa6cb85719d00b3add82872bae08e1547e2d333deb6d0362bd4157d"}
Oct 30 00:42:55.254878 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:42:55.254823    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-ljbgk-w44n4" podStartSLOduration=2.25480751 podStartE2EDuration="2.25480751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:55.25417975 +0000 UTC m=+87.592281673" watchObservedRunningTime="2026-10-30 00:42:55.25480751 +0000 UTC m=+87.592909421"
Oct 30 00:43:19.037775 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:43:19.037734    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:43:49.037707 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:43:49.037633    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:44:18.508007 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:44:18.507969    4864 generic.go:356] "Generic (PLEG): container finished" podID="829d9860-08f5-4ccf-8ba6-d53907fc44b6" containerID="d88730b8e6560680fc47c67c989d4ab056e3a6a5fbebc8c7aa7a4c4fb41fdff9" exitCode=0
Oct 30 00:44:18.508434 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:44:18.508014    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ljbgk-w44n4" event={"ID":"829d9860-08f5-4ccf-8ba6-d53907fc44b6","Type":"ContainerDied","Data":"d88730b8e6560680fc47c67c989d4ab056e3a6a5fbebc8c7aa7a4c4fb41fdff9"}
Oct 30 00:44:19.037577 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:44:19.037540    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:44:19.631267 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:44:19.631241    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ljbgk-w44n4"
Oct 30 00:44:19.748289 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:44:19.748256    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/829d9860-08f5-4ccf-8ba6-d53907fc44b6-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/829d9860-08f5-4ccf-8ba6-d53907fc44b6-service-ca-bundle\") pod \"829d9860-08f5-4ccf-8ba6-d53907fc44b6\" (UID: \"829d9860-08f5-4ccf-8ba6-d53907fc44b6\") "
Oct 30 00:44:19.748462 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:44:19.748334    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/829d9860-08f5-4ccf-8ba6-d53907fc44b6-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/829d9860-08f5-4ccf-8ba6-d53907fc44b6-trusted-ca-bundle\") pod \"829d9860-08f5-4ccf-8ba6-d53907fc44b6\" (UID: \"829d9860-08f5-4ccf-8ba6-d53907fc44b6\") "
Oct 30 00:44:19.748462 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:44:19.748424    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/829d9860-08f5-4ccf-8ba6-d53907fc44b6-kube-api-access-jxns4\" (UniqueName: \"kubernetes.io/projected/829d9860-08f5-4ccf-8ba6-d53907fc44b6-kube-api-access-jxns4\") pod \"829d9860-08f5-4ccf-8ba6-d53907fc44b6\" (UID: \"829d9860-08f5-4ccf-8ba6-d53907fc44b6\") "
Oct 30 00:44:19.748585 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:44:19.748463    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/829d9860-08f5-4ccf-8ba6-d53907fc44b6-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/829d9860-08f5-4ccf-8ba6-d53907fc44b6-archives-path\") pod \"829d9860-08f5-4ccf-8ba6-d53907fc44b6\" (UID: \"829d9860-08f5-4ccf-8ba6-d53907fc44b6\") "
Oct 30 00:44:19.748658 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:44:19.748636    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/829d9860-08f5-4ccf-8ba6-d53907fc44b6-service-ca-bundle" pod "829d9860-08f5-4ccf-8ba6-d53907fc44b6" (UID: "829d9860-08f5-4ccf-8ba6-d53907fc44b6"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:44:19.748718 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:44:19.748652    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/829d9860-08f5-4ccf-8ba6-d53907fc44b6-trusted-ca-bundle" pod "829d9860-08f5-4ccf-8ba6-d53907fc44b6" (UID: "829d9860-08f5-4ccf-8ba6-d53907fc44b6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:44:19.748846 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:44:19.748824    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829d9860-08f5-4ccf-8ba6-d53907fc44b6-archives-path" pod "829d9860-08f5-4ccf-8ba6-d53907fc44b6" (UID: "829d9860-08f5-4ccf-8ba6-d53907fc44b6"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:44:19.750630 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:44:19.750607    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/829d9860-08f5-4ccf-8ba6-d53907fc44b6-kube-api-access-jxns4" pod "829d9860-08f5-4ccf-8ba6-d53907fc44b6" (UID: "829d9860-08f5-4ccf-8ba6-d53907fc44b6"). InnerVolumeSpecName "kube-api-access-jxns4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:44:19.849789 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:44:19.849752    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxns4\" (UniqueName: \"kubernetes.io/projected/829d9860-08f5-4ccf-8ba6-d53907fc44b6-kube-api-access-jxns4\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:44:19.849789 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:44:19.849784    4864 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/829d9860-08f5-4ccf-8ba6-d53907fc44b6-archives-path\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:44:19.849789 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:44:19.849793    4864 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/829d9860-08f5-4ccf-8ba6-d53907fc44b6-service-ca-bundle\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:44:19.850019 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:44:19.849802    4864 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/829d9860-08f5-4ccf-8ba6-d53907fc44b6-trusted-ca-bundle\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:44:20.515991 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:44:20.515961    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ljbgk-w44n4"
Oct 30 00:44:20.515991 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:44:20.515978    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ljbgk-w44n4" event={"ID":"829d9860-08f5-4ccf-8ba6-d53907fc44b6","Type":"ContainerDied","Data":"0960c9eb6fa6cb85719d00b3add82872bae08e1547e2d333deb6d0362bd4157d"}
Oct 30 00:44:20.516217 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:44:20.516018    4864 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0960c9eb6fa6cb85719d00b3add82872bae08e1547e2d333deb6d0362bd4157d"
Oct 30 00:44:49.037790 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:44:49.037746    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:45:19.037621 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:45:19.037543    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:45:35.076795 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.076757    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v"]
Oct 30 00:45:35.077254 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.077061    4864 cpu_manager.go:414] "containerMap: removing container" podUID="829d9860-08f5-4ccf-8ba6-d53907fc44b6" containerName="insights-gathering"
Oct 30 00:45:35.077254 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.077086    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="829d9860-08f5-4ccf-8ba6-d53907fc44b6" containerName="insights-gathering"
Oct 30 00:45:35.077254 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.077158    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="829d9860-08f5-4ccf-8ba6-d53907fc44b6" containerName="insights-gathering"
Oct 30 00:45:35.080235 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.080214    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v"
Oct 30 00:45:35.103942 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.103920    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 00:45:35.103942 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.103930    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-wckrn\""
Oct 30 00:45:35.103942 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.103920    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 00:45:35.105577 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.105530    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v"]
Oct 30 00:45:35.173656 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.173619    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7"]
Oct 30 00:45:35.179469 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.179443    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7"
Oct 30 00:45:35.183166 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.183137    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v\" (UID: \"fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v"
Oct 30 00:45:35.183279 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.183177    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v\" (UID: \"fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v"
Oct 30 00:45:35.183279 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.183213    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j7k9\" (UniqueName: \"kubernetes.io/projected/fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4-kube-api-access-4j7k9\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v\" (UID: \"fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v"
Oct 30 00:45:35.201321 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.201296    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-qghb7\""
Oct 30 00:45:35.202335 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.202312    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7"]
Oct 30 00:45:35.284616 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.284585    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3ab963c-9d14-4134-ad4a-5334e3507c9f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7\" (UID: \"d3ab963c-9d14-4134-ad4a-5334e3507c9f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7"
Oct 30 00:45:35.284788 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.284636    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v\" (UID: \"fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v"
Oct 30 00:45:35.284788 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.284705    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v\" (UID: \"fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v"
Oct 30 00:45:35.284788 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.284749    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-259xb\" (UniqueName: \"kubernetes.io/projected/d3ab963c-9d14-4134-ad4a-5334e3507c9f-kube-api-access-259xb\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7\" (UID: \"d3ab963c-9d14-4134-ad4a-5334e3507c9f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7"
Oct 30 00:45:35.284940 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.284788    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j7k9\" (UniqueName: \"kubernetes.io/projected/fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4-kube-api-access-4j7k9\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v\" (UID: \"fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v"
Oct 30 00:45:35.284940 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.284826    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3ab963c-9d14-4134-ad4a-5334e3507c9f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7\" (UID: \"d3ab963c-9d14-4134-ad4a-5334e3507c9f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7"
Oct 30 00:45:35.285034 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.285011    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v\" (UID: \"fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v"
Oct 30 00:45:35.285090 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.285038    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v\" (UID: \"fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v"
Oct 30 00:45:35.306253 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.306226    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j7k9\" (UniqueName: \"kubernetes.io/projected/fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4-kube-api-access-4j7k9\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v\" (UID: \"fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v"
Oct 30 00:45:35.385427 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.385346    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-259xb\" (UniqueName: \"kubernetes.io/projected/d3ab963c-9d14-4134-ad4a-5334e3507c9f-kube-api-access-259xb\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7\" (UID: \"d3ab963c-9d14-4134-ad4a-5334e3507c9f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7"
Oct 30 00:45:35.385427 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.385396    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3ab963c-9d14-4134-ad4a-5334e3507c9f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7\" (UID: \"d3ab963c-9d14-4134-ad4a-5334e3507c9f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7"
Oct 30 00:45:35.385427 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.385426    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3ab963c-9d14-4134-ad4a-5334e3507c9f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7\" (UID: \"d3ab963c-9d14-4134-ad4a-5334e3507c9f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7"
Oct 30 00:45:35.385829 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.385806    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3ab963c-9d14-4134-ad4a-5334e3507c9f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7\" (UID: \"d3ab963c-9d14-4134-ad4a-5334e3507c9f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7"
Oct 30 00:45:35.385897 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.385835    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3ab963c-9d14-4134-ad4a-5334e3507c9f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7\" (UID: \"d3ab963c-9d14-4134-ad4a-5334e3507c9f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7"
Oct 30 00:45:35.395885 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.395867    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v"
Oct 30 00:45:35.411308 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.411282    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-259xb\" (UniqueName: \"kubernetes.io/projected/d3ab963c-9d14-4134-ad4a-5334e3507c9f-kube-api-access-259xb\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7\" (UID: \"d3ab963c-9d14-4134-ad4a-5334e3507c9f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7"
Oct 30 00:45:35.494131 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.494096    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7"
Oct 30 00:45:35.693684 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.693600    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v"]
Oct 30 00:45:35.698948 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:45:35.698919    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbcdee40_58d0_46e6_a1c5_29d5a0e9c4b4.slice/crio-a640004f023273be80718d1680c51f1819bd92e487bfeb42c832dab7c5da2431 WatchSource:0}: Error finding container a640004f023273be80718d1680c51f1819bd92e487bfeb42c832dab7c5da2431: Status 404 returned error can't find the container with id a640004f023273be80718d1680c51f1819bd92e487bfeb42c832dab7c5da2431
Oct 30 00:45:35.778795 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.778762    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"]
Oct 30 00:45:35.783041 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.783014    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v" event={"ID":"fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4","Type":"ContainerStarted","Data":"a640004f023273be80718d1680c51f1819bd92e487bfeb42c832dab7c5da2431"}
Oct 30 00:45:35.783165 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.783123    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:35.790986 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.790856    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7"]
Oct 30 00:45:35.794127 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:45:35.794106    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ab963c_9d14_4134_ad4a_5334e3507c9f.slice/crio-003f6ab12b1a1faed0126e17b6bebf4be40cad0edebbb43a2cc7c489d08da156 WatchSource:0}: Error finding container 003f6ab12b1a1faed0126e17b6bebf4be40cad0edebbb43a2cc7c489d08da156: Status 404 returned error can't find the container with id 003f6ab12b1a1faed0126e17b6bebf4be40cad0edebbb43a2cc7c489d08da156
Oct 30 00:45:35.795869 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.795853    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 00:45:35.795952 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.795874    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 00:45:35.795952 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.795944    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 00:45:35.796019 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.795993    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-htf9g\""
Oct 30 00:45:35.796450 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.796434    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 00:45:35.808698 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.808678    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"]
Oct 30 00:45:35.889352 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.889321    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-c7wlt\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:35.889352 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.889353    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9651f3a-0816-4f3a-8792-2951ba4bc28a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-c7wlt\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:35.889557 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.889371    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m5v2\" (UniqueName: \"kubernetes.io/projected/e9651f3a-0816-4f3a-8792-2951ba4bc28a-kube-api-access-8m5v2\") pod \"istiod-openshift-gateway-7876bfcb74-c7wlt\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:35.889557 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.889391    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-c7wlt\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:35.889557 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.889488    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-c7wlt\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:35.889557 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.889526    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-c7wlt\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:35.889699 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.889605    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9651f3a-0816-4f3a-8792-2951ba4bc28a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-c7wlt\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:35.990474 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.990437    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-c7wlt\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:35.990664 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.990478    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-c7wlt\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:35.990664 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.990512    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-c7wlt\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:35.990664 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.990560    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9651f3a-0816-4f3a-8792-2951ba4bc28a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-c7wlt\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:35.990664 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.990623    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-c7wlt\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:35.990664 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.990646    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9651f3a-0816-4f3a-8792-2951ba4bc28a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-c7wlt\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:35.990932 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.990673    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m5v2\" (UniqueName: \"kubernetes.io/projected/e9651f3a-0816-4f3a-8792-2951ba4bc28a-kube-api-access-8m5v2\") pod \"istiod-openshift-gateway-7876bfcb74-c7wlt\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:35.991278 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.991256    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-c7wlt\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:35.992969 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.992948    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9651f3a-0816-4f3a-8792-2951ba4bc28a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-c7wlt\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:35.993082 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.992998    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-c7wlt\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:35.993388 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.993366    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-c7wlt\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:35.993427 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:35.993375    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9651f3a-0816-4f3a-8792-2951ba4bc28a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-c7wlt\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:36.012017 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:36.011995    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-c7wlt\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:36.012109 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:36.012040    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m5v2\" (UniqueName: \"kubernetes.io/projected/e9651f3a-0816-4f3a-8792-2951ba4bc28a-kube-api-access-8m5v2\") pod \"istiod-openshift-gateway-7876bfcb74-c7wlt\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:36.096366 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:36.096332    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:36.407726 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:36.407649    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"]
Oct 30 00:45:36.415127 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:45:36.415094    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9651f3a_0816_4f3a_8792_2951ba4bc28a.slice/crio-6460b790ab339d32924e792bb3dc3df42d39de41ae1d352df63d65889d1f8ac2 WatchSource:0}: Error finding container 6460b790ab339d32924e792bb3dc3df42d39de41ae1d352df63d65889d1f8ac2: Status 404 returned error can't find the container with id 6460b790ab339d32924e792bb3dc3df42d39de41ae1d352df63d65889d1f8ac2
Oct 30 00:45:36.789174 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:36.789137    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt" event={"ID":"e9651f3a-0816-4f3a-8792-2951ba4bc28a","Type":"ContainerStarted","Data":"6460b790ab339d32924e792bb3dc3df42d39de41ae1d352df63d65889d1f8ac2"}
Oct 30 00:45:36.790769 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:36.790734    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7" event={"ID":"d3ab963c-9d14-4134-ad4a-5334e3507c9f","Type":"ContainerStarted","Data":"003f6ab12b1a1faed0126e17b6bebf4be40cad0edebbb43a2cc7c489d08da156"}
Oct 30 00:45:39.820179 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:39.820141    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:45:39.820562 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:39.820216    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:45:40.816725 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:40.815947    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt" event={"ID":"e9651f3a-0816-4f3a-8792-2951ba4bc28a","Type":"ContainerStarted","Data":"8065bc590afe6b0336ba260dbdf3f758336319aa10d2d04b5ae58f33228a741d"}
Oct 30 00:45:40.816725 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:40.816544    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:40.818841 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:40.818547    4864 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-c7wlt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 00:45:40.818841 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:40.818606    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt" podUID="e9651f3a-0816-4f3a-8792-2951ba4bc28a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 00:45:40.889452 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:40.889401    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt" podStartSLOduration=2.487951277 podStartE2EDuration="5.889383563s" totalImagesPullingTime="3.401432286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:40.878765574 +0000 UTC m=+253.216867521" watchObservedRunningTime="2026-10-30 00:45:40.889383563 +0000 UTC m=+253.227485475"
Oct 30 00:45:41.821327 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:41.821116    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:45:42.826516 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:42.826479    4864 generic.go:356] "Generic (PLEG): container finished" podID="fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4" containerID="6b6c0f1bb60ce4e09049a3179185ea4def8004cab1d9fb2a56fd96e98fefe763" exitCode=0
Oct 30 00:45:42.826966 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:42.826567    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v" event={"ID":"fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4","Type":"ContainerDied","Data":"6b6c0f1bb60ce4e09049a3179185ea4def8004cab1d9fb2a56fd96e98fefe763"}
Oct 30 00:45:42.828017 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:42.827997    4864 generic.go:356] "Generic (PLEG): container finished" podID="d3ab963c-9d14-4134-ad4a-5334e3507c9f" containerID="b8c41b36a8a65c895a26098f8925fb6a2171a1138c99ab86d21c7304eee1342b" exitCode=0
Oct 30 00:45:42.828126 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:42.828100    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7" event={"ID":"d3ab963c-9d14-4134-ad4a-5334e3507c9f","Type":"ContainerDied","Data":"b8c41b36a8a65c895a26098f8925fb6a2171a1138c99ab86d21c7304eee1342b"}
Oct 30 00:45:45.846250 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:45.846173    4864 generic.go:356] "Generic (PLEG): container finished" podID="fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4" containerID="bf8d797e6143384eb585f8ea623ecd3137f312142d4e036bcd3b1cb4816758ad" exitCode=0
Oct 30 00:45:45.846664 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:45.846257    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v" event={"ID":"fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4","Type":"ContainerDied","Data":"bf8d797e6143384eb585f8ea623ecd3137f312142d4e036bcd3b1cb4816758ad"}
Oct 30 00:45:45.848095 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:45.848043    4864 generic.go:356] "Generic (PLEG): container finished" podID="d3ab963c-9d14-4134-ad4a-5334e3507c9f" containerID="42556e7ed78cc5f053d21376c84ffde311ef16b56b649c2edbda8cd2ea17b470" exitCode=0
Oct 30 00:45:45.848164 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:45.848112    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7" event={"ID":"d3ab963c-9d14-4134-ad4a-5334e3507c9f","Type":"ContainerDied","Data":"42556e7ed78cc5f053d21376c84ffde311ef16b56b649c2edbda8cd2ea17b470"}
Oct 30 00:45:49.045135 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:45:49.045101    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:45:52.882684 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:52.882595    4864 generic.go:356] "Generic (PLEG): container finished" podID="fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4" containerID="82a722cce384a13c0bc8238a3a8488643ae921495711fd5e8ac3ae6dd474e11d" exitCode=0
Oct 30 00:45:52.882684 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:52.882671    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v" event={"ID":"fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4","Type":"ContainerDied","Data":"82a722cce384a13c0bc8238a3a8488643ae921495711fd5e8ac3ae6dd474e11d"}
Oct 30 00:45:52.884766 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:52.884740    4864 generic.go:356] "Generic (PLEG): container finished" podID="d3ab963c-9d14-4134-ad4a-5334e3507c9f" containerID="80a0c7a3ac44bdec1cf827ed8b8a75d592e19463637ec0dec3a06b82a2a4762c" exitCode=0
Oct 30 00:45:52.884867 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:52.884782    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7" event={"ID":"d3ab963c-9d14-4134-ad4a-5334e3507c9f","Type":"ContainerDied","Data":"80a0c7a3ac44bdec1cf827ed8b8a75d592e19463637ec0dec3a06b82a2a4762c"}
Oct 30 00:45:54.034389 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.034366    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7"
Oct 30 00:45:54.037143 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.037123    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v"
Oct 30 00:45:54.054866 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.054842    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4-kube-api-access-4j7k9\" (UniqueName: \"kubernetes.io/projected/fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4-kube-api-access-4j7k9\") pod \"fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4\" (UID: \"fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4\") "
Oct 30 00:45:54.054976 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.054895    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ab963c-9d14-4134-ad4a-5334e3507c9f-kube-api-access-259xb\" (UniqueName: \"kubernetes.io/projected/d3ab963c-9d14-4134-ad4a-5334e3507c9f-kube-api-access-259xb\") pod \"d3ab963c-9d14-4134-ad4a-5334e3507c9f\" (UID: \"d3ab963c-9d14-4134-ad4a-5334e3507c9f\") "
Oct 30 00:45:54.054976 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.054924    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4-bundle\") pod \"fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4\" (UID: \"fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4\") "
Oct 30 00:45:54.054976 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.054945    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ab963c-9d14-4134-ad4a-5334e3507c9f-util\" (UniqueName: \"kubernetes.io/empty-dir/d3ab963c-9d14-4134-ad4a-5334e3507c9f-util\") pod \"d3ab963c-9d14-4134-ad4a-5334e3507c9f\" (UID: \"d3ab963c-9d14-4134-ad4a-5334e3507c9f\") "
Oct 30 00:45:54.054976 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.054970    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4-util\" (UniqueName: \"kubernetes.io/empty-dir/fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4-util\") pod \"fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4\" (UID: \"fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4\") "
Oct 30 00:45:54.055216 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.054993    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ab963c-9d14-4134-ad4a-5334e3507c9f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3ab963c-9d14-4134-ad4a-5334e3507c9f-bundle\") pod \"d3ab963c-9d14-4134-ad4a-5334e3507c9f\" (UID: \"d3ab963c-9d14-4134-ad4a-5334e3507c9f\") "
Oct 30 00:45:54.055432 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.055401    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4-bundle" pod "fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4" (UID: "fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:45:54.055714 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.055689    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ab963c-9d14-4134-ad4a-5334e3507c9f-bundle" pod "d3ab963c-9d14-4134-ad4a-5334e3507c9f" (UID: "d3ab963c-9d14-4134-ad4a-5334e3507c9f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:45:54.057669 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.057647    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4-kube-api-access-4j7k9" pod "fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4" (UID: "fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4"). InnerVolumeSpecName "kube-api-access-4j7k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:45:54.057753 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.057719    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ab963c-9d14-4134-ad4a-5334e3507c9f-kube-api-access-259xb" pod "d3ab963c-9d14-4134-ad4a-5334e3507c9f" (UID: "d3ab963c-9d14-4134-ad4a-5334e3507c9f"). InnerVolumeSpecName "kube-api-access-259xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:45:54.059537 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.059514    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ab963c-9d14-4134-ad4a-5334e3507c9f-util" pod "d3ab963c-9d14-4134-ad4a-5334e3507c9f" (UID: "d3ab963c-9d14-4134-ad4a-5334e3507c9f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:45:54.060559 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.060525    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4-util" pod "fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4" (UID: "fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:45:54.156290 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.156212    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-259xb\" (UniqueName: \"kubernetes.io/projected/d3ab963c-9d14-4134-ad4a-5334e3507c9f-kube-api-access-259xb\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:45:54.156290 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.156239    4864 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4-bundle\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:45:54.156290 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.156250    4864 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3ab963c-9d14-4134-ad4a-5334e3507c9f-util\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:45:54.156290 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.156258    4864 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4-util\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:45:54.156290 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.156266    4864 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3ab963c-9d14-4134-ad4a-5334e3507c9f-bundle\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:45:54.156290 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.156274    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j7k9\" (UniqueName: \"kubernetes.io/projected/fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4-kube-api-access-4j7k9\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:45:54.894151 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.894104    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v" event={"ID":"fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4","Type":"ContainerDied","Data":"a640004f023273be80718d1680c51f1819bd92e487bfeb42c832dab7c5da2431"}
Oct 30 00:45:54.894350 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.894169    4864 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a640004f023273be80718d1680c51f1819bd92e487bfeb42c832dab7c5da2431"
Oct 30 00:45:54.894350 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.894122    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nw97v"
Oct 30 00:45:54.896041 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.896019    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7"
Oct 30 00:45:54.896168 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.896040    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqx6b7" event={"ID":"d3ab963c-9d14-4134-ad4a-5334e3507c9f","Type":"ContainerDied","Data":"003f6ab12b1a1faed0126e17b6bebf4be40cad0edebbb43a2cc7c489d08da156"}
Oct 30 00:45:54.896168 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:45:54.896088    4864 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="003f6ab12b1a1faed0126e17b6bebf4be40cad0edebbb43a2cc7c489d08da156"
Oct 30 00:46:17.383300 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.383265    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"]
Oct 30 00:46:17.383705 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.383561    4864 cpu_manager.go:414] "containerMap: removing container" podUID="d3ab963c-9d14-4134-ad4a-5334e3507c9f" containerName="pull"
Oct 30 00:46:17.383705 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.383571    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ab963c-9d14-4134-ad4a-5334e3507c9f" containerName="pull"
Oct 30 00:46:17.383705 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.383581    4864 cpu_manager.go:414] "containerMap: removing container" podUID="fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4" containerName="pull"
Oct 30 00:46:17.383705 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.383586    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4" containerName="pull"
Oct 30 00:46:17.383705 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.383596    4864 cpu_manager.go:414] "containerMap: removing container" podUID="d3ab963c-9d14-4134-ad4a-5334e3507c9f" containerName="util"
Oct 30 00:46:17.383705 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.383601    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ab963c-9d14-4134-ad4a-5334e3507c9f" containerName="util"
Oct 30 00:46:17.383705 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.383611    4864 cpu_manager.go:414] "containerMap: removing container" podUID="d3ab963c-9d14-4134-ad4a-5334e3507c9f" containerName="extract"
Oct 30 00:46:17.383705 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.383616    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ab963c-9d14-4134-ad4a-5334e3507c9f" containerName="extract"
Oct 30 00:46:17.383705 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.383625    4864 cpu_manager.go:414] "containerMap: removing container" podUID="fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4" containerName="util"
Oct 30 00:46:17.383705 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.383630    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4" containerName="util"
Oct 30 00:46:17.383705 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.383644    4864 cpu_manager.go:414] "containerMap: removing container" podUID="fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4" containerName="extract"
Oct 30 00:46:17.383705 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.383648    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4" containerName="extract"
Oct 30 00:46:17.383705 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.383694    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbcdee40-58d0-46e6-a1c5-29d5a0e9c4b4" containerName="extract"
Oct 30 00:46:17.383705 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.383705    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ab963c-9d14-4134-ad4a-5334e3507c9f" containerName="extract"
Oct 30 00:46:17.385850 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.385785    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"
Oct 30 00:46:17.399236 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.399213    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 00:46:17.400436 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.400418    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-lkrjc\""
Oct 30 00:46:17.400483 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.400464    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 00:46:17.400521 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.400465    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 00:46:17.410493 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.410476    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 00:46:17.410540 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.410479    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 00:46:17.423842 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.423821    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6c2c6cca-8f41-45a9-a38d-1bace487c94b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-gwnjx\" (UID: \"6c2c6cca-8f41-45a9-a38d-1bace487c94b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"
Oct 30 00:46:17.423951 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.423848    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c2c6cca-8f41-45a9-a38d-1bace487c94b-metrics-cert\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-gwnjx\" (UID: \"6c2c6cca-8f41-45a9-a38d-1bace487c94b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"
Oct 30 00:46:17.423951 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.423864    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6cca-8f41-45a9-a38d-1bace487c94b-manifests\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-gwnjx\" (UID: \"6c2c6cca-8f41-45a9-a38d-1bace487c94b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"
Oct 30 00:46:17.423951 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.423894    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6cca-8f41-45a9-a38d-1bace487c94b-tmp\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-gwnjx\" (UID: \"6c2c6cca-8f41-45a9-a38d-1bace487c94b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"
Oct 30 00:46:17.424048 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.424012    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrr54\" (UniqueName: \"kubernetes.io/projected/6c2c6cca-8f41-45a9-a38d-1bace487c94b-kube-api-access-nrr54\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-gwnjx\" (UID: \"6c2c6cca-8f41-45a9-a38d-1bace487c94b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"
Oct 30 00:46:17.424105 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.424059    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6c2c6cca-8f41-45a9-a38d-1bace487c94b-webhook-cert\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-gwnjx\" (UID: \"6c2c6cca-8f41-45a9-a38d-1bace487c94b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"
Oct 30 00:46:17.446920 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.446894    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"]
Oct 30 00:46:17.525227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.525198    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6cca-8f41-45a9-a38d-1bace487c94b-tmp\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-gwnjx\" (UID: \"6c2c6cca-8f41-45a9-a38d-1bace487c94b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"
Oct 30 00:46:17.525401 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.525235    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrr54\" (UniqueName: \"kubernetes.io/projected/6c2c6cca-8f41-45a9-a38d-1bace487c94b-kube-api-access-nrr54\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-gwnjx\" (UID: \"6c2c6cca-8f41-45a9-a38d-1bace487c94b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"
Oct 30 00:46:17.525401 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.525264    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6c2c6cca-8f41-45a9-a38d-1bace487c94b-webhook-cert\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-gwnjx\" (UID: \"6c2c6cca-8f41-45a9-a38d-1bace487c94b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"
Oct 30 00:46:17.525401 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.525302    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6c2c6cca-8f41-45a9-a38d-1bace487c94b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-gwnjx\" (UID: \"6c2c6cca-8f41-45a9-a38d-1bace487c94b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"
Oct 30 00:46:17.525401 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.525317    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c2c6cca-8f41-45a9-a38d-1bace487c94b-metrics-cert\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-gwnjx\" (UID: \"6c2c6cca-8f41-45a9-a38d-1bace487c94b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"
Oct 30 00:46:17.525401 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.525333    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6cca-8f41-45a9-a38d-1bace487c94b-manifests\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-gwnjx\" (UID: \"6c2c6cca-8f41-45a9-a38d-1bace487c94b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"
Oct 30 00:46:17.525703 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.525673    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6cca-8f41-45a9-a38d-1bace487c94b-manifests\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-gwnjx\" (UID: \"6c2c6cca-8f41-45a9-a38d-1bace487c94b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"
Oct 30 00:46:17.527700 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.527676    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6cca-8f41-45a9-a38d-1bace487c94b-tmp\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-gwnjx\" (UID: \"6c2c6cca-8f41-45a9-a38d-1bace487c94b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"
Oct 30 00:46:17.527797 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.527756    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6c2c6cca-8f41-45a9-a38d-1bace487c94b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-gwnjx\" (UID: \"6c2c6cca-8f41-45a9-a38d-1bace487c94b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"
Oct 30 00:46:17.527905 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.527889    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c2c6cca-8f41-45a9-a38d-1bace487c94b-metrics-cert\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-gwnjx\" (UID: \"6c2c6cca-8f41-45a9-a38d-1bace487c94b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"
Oct 30 00:46:17.527939 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.527889    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6c2c6cca-8f41-45a9-a38d-1bace487c94b-webhook-cert\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-gwnjx\" (UID: \"6c2c6cca-8f41-45a9-a38d-1bace487c94b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"
Oct 30 00:46:17.547130 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.547099    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrr54\" (UniqueName: \"kubernetes.io/projected/6c2c6cca-8f41-45a9-a38d-1bace487c94b-kube-api-access-nrr54\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-gwnjx\" (UID: \"6c2c6cca-8f41-45a9-a38d-1bace487c94b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"
Oct 30 00:46:17.699488 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:17.699395    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"
Oct 30 00:46:18.038973 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:18.038939    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"]
Oct 30 00:46:18.043730 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:46:18.043697    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c2c6cca_8f41_45a9_a38d_1bace487c94b.slice/crio-dfb041b6fc78bf9b15e6f7882fb56c709e907d144ef159eabeb7d8574e9085ee WatchSource:0}: Error finding container dfb041b6fc78bf9b15e6f7882fb56c709e907d144ef159eabeb7d8574e9085ee: Status 404 returned error can't find the container with id dfb041b6fc78bf9b15e6f7882fb56c709e907d144ef159eabeb7d8574e9085ee
Oct 30 00:46:18.989930 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:18.989887    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx" event={"ID":"6c2c6cca-8f41-45a9-a38d-1bace487c94b","Type":"ContainerStarted","Data":"dfb041b6fc78bf9b15e6f7882fb56c709e907d144ef159eabeb7d8574e9085ee"}
Oct 30 00:46:19.048013 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:46:19.047971    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:46:21.000803 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:21.000717    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx" event={"ID":"6c2c6cca-8f41-45a9-a38d-1bace487c94b","Type":"ContainerStarted","Data":"6a47efc85a8341ac4d1128b6039523c75cc9ef93f3e699b877e04a4dc80d4508"}
Oct 30 00:46:22.011462 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:22.011427    4864 generic.go:356] "Generic (PLEG): container finished" podID="6c2c6cca-8f41-45a9-a38d-1bace487c94b" containerID="6a47efc85a8341ac4d1128b6039523c75cc9ef93f3e699b877e04a4dc80d4508" exitCode=0
Oct 30 00:46:22.011925 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:22.011518    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx" event={"ID":"6c2c6cca-8f41-45a9-a38d-1bace487c94b","Type":"ContainerDied","Data":"6a47efc85a8341ac4d1128b6039523c75cc9ef93f3e699b877e04a4dc80d4508"}
Oct 30 00:46:23.016657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:23.016618    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx" event={"ID":"6c2c6cca-8f41-45a9-a38d-1bace487c94b","Type":"ContainerStarted","Data":"8bdbfeb50f085d58deb8ba8b11ba5daeb728ceaf8935356d66f07e73cb07259f"}
Oct 30 00:46:23.017132 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:23.016744    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"
Oct 30 00:46:23.096977 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:23.096924    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx" podStartSLOduration=2.599123385 podStartE2EDuration="6.096909369s" totalImagesPullingTime="3.005101109s" totalInitContainerRuntime="492.684875ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:23.08590854 +0000 UTC m=+295.424010450" watchObservedRunningTime="2026-10-30 00:46:23.096909369 +0000 UTC m=+295.435011273"
Oct 30 00:46:27.808776 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:27.808748    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 00:46:27.809221 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:27.808963    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 00:46:34.022770 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:34.022734    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-gwnjx"
Oct 30 00:46:47.577923 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:47.577840    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-57cd69bbcd-76gbt"]
Oct 30 00:46:47.581261 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:47.581240    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-76gbt"
Oct 30 00:46:47.596171 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:47.596149    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 00:46:47.597484 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:47.597463    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-t6svh\""
Oct 30 00:46:47.600135 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:47.600119    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 00:46:47.611110 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:47.611064    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57cd69bbcd-76gbt"]
Oct 30 00:46:47.669630 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:47.669594    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bd30c09-5438-468b-b353-eb0a1ba483b5-tls-certs\") pod \"kube-auth-proxy-57cd69bbcd-76gbt\" (UID: \"6bd30c09-5438-468b-b353-eb0a1ba483b5\") " pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-76gbt"
Oct 30 00:46:47.669630 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:47.669629    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bd30c09-5438-468b-b353-eb0a1ba483b5-tmp\") pod \"kube-auth-proxy-57cd69bbcd-76gbt\" (UID: \"6bd30c09-5438-468b-b353-eb0a1ba483b5\") " pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-76gbt"
Oct 30 00:46:47.669828 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:47.669726    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqmfk\" (UniqueName: \"kubernetes.io/projected/6bd30c09-5438-468b-b353-eb0a1ba483b5-kube-api-access-nqmfk\") pod \"kube-auth-proxy-57cd69bbcd-76gbt\" (UID: \"6bd30c09-5438-468b-b353-eb0a1ba483b5\") " pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-76gbt"
Oct 30 00:46:47.770286 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:47.770254    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bd30c09-5438-468b-b353-eb0a1ba483b5-tls-certs\") pod \"kube-auth-proxy-57cd69bbcd-76gbt\" (UID: \"6bd30c09-5438-468b-b353-eb0a1ba483b5\") " pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-76gbt"
Oct 30 00:46:47.770286 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:47.770292    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bd30c09-5438-468b-b353-eb0a1ba483b5-tmp\") pod \"kube-auth-proxy-57cd69bbcd-76gbt\" (UID: \"6bd30c09-5438-468b-b353-eb0a1ba483b5\") " pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-76gbt"
Oct 30 00:46:47.770541 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:47.770341    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqmfk\" (UniqueName: \"kubernetes.io/projected/6bd30c09-5438-468b-b353-eb0a1ba483b5-kube-api-access-nqmfk\") pod \"kube-auth-proxy-57cd69bbcd-76gbt\" (UID: \"6bd30c09-5438-468b-b353-eb0a1ba483b5\") " pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-76gbt"
Oct 30 00:46:47.772581 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:47.772551    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bd30c09-5438-468b-b353-eb0a1ba483b5-tmp\") pod \"kube-auth-proxy-57cd69bbcd-76gbt\" (UID: \"6bd30c09-5438-468b-b353-eb0a1ba483b5\") " pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-76gbt"
Oct 30 00:46:47.772894 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:47.772872    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bd30c09-5438-468b-b353-eb0a1ba483b5-tls-certs\") pod \"kube-auth-proxy-57cd69bbcd-76gbt\" (UID: \"6bd30c09-5438-468b-b353-eb0a1ba483b5\") " pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-76gbt"
Oct 30 00:46:47.793042 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:47.793010    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqmfk\" (UniqueName: \"kubernetes.io/projected/6bd30c09-5438-468b-b353-eb0a1ba483b5-kube-api-access-nqmfk\") pod \"kube-auth-proxy-57cd69bbcd-76gbt\" (UID: \"6bd30c09-5438-468b-b353-eb0a1ba483b5\") " pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-76gbt"
Oct 30 00:46:47.893161 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:47.893058    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-76gbt"
Oct 30 00:46:48.188674 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:48.188591    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57cd69bbcd-76gbt"]
Oct 30 00:46:48.193483 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:46:48.193455    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd30c09_5438_468b_b353_eb0a1ba483b5.slice/crio-ab9338604ec0b6c1e30b22259b4d8d991eb984e53606150157a391087ff41975 WatchSource:0}: Error finding container ab9338604ec0b6c1e30b22259b4d8d991eb984e53606150157a391087ff41975: Status 404 returned error can't find the container with id ab9338604ec0b6c1e30b22259b4d8d991eb984e53606150157a391087ff41975
Oct 30 00:46:48.195448 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:48.195432    4864 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:46:49.047701 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:46:49.047660    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:46:49.123797 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:49.123753    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-76gbt" event={"ID":"6bd30c09-5438-468b-b353-eb0a1ba483b5","Type":"ContainerStarted","Data":"ab9338604ec0b6c1e30b22259b4d8d991eb984e53606150157a391087ff41975"}
Oct 30 00:46:51.134149 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:51.134104    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-76gbt" event={"ID":"6bd30c09-5438-468b-b353-eb0a1ba483b5","Type":"ContainerStarted","Data":"a8eae57ecbc795d302aec31d49004e0ef10ae19f9746514d61fcf5050fbd4684"}
Oct 30 00:46:51.172930 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:51.172875    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-76gbt" podStartSLOduration=1.938187284 podStartE2EDuration="4.172859866s" totalImagesPullingTime="2.234672582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:51.171233267 +0000 UTC m=+323.509335178" watchObservedRunningTime="2026-10-30 00:46:51.172859866 +0000 UTC m=+323.510961776"
Oct 30 00:46:55.724330 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:55.724294    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-nbrcn"]
Oct 30 00:46:55.732813 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:55.732791    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nbrcn"
Oct 30 00:46:55.746604 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:55.746578    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 00:46:55.746722 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:55.746705    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 00:46:55.779138 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:55.779106    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-vwd5x\""
Oct 30 00:46:55.779417 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:55.779395    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 00:46:55.780485 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:55.780467    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-nbrcn"]
Oct 30 00:46:55.844455 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:55.844414    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/145c8ccc-6041-44e5-8df0-c8c2420dfff2-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-nbrcn\" (UID: \"145c8ccc-6041-44e5-8df0-c8c2420dfff2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nbrcn"
Oct 30 00:46:55.844624 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:55.844465    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/145c8ccc-6041-44e5-8df0-c8c2420dfff2-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-nbrcn\" (UID: \"145c8ccc-6041-44e5-8df0-c8c2420dfff2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nbrcn"
Oct 30 00:46:55.844624 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:55.844593    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/145c8ccc-6041-44e5-8df0-c8c2420dfff2-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-nbrcn\" (UID: \"145c8ccc-6041-44e5-8df0-c8c2420dfff2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nbrcn"
Oct 30 00:46:55.844723 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:55.844641    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npbcg\" (UniqueName: \"kubernetes.io/projected/145c8ccc-6041-44e5-8df0-c8c2420dfff2-kube-api-access-npbcg\") pod \"kserve-module-controller-manager-7566cf5fc8-nbrcn\" (UID: \"145c8ccc-6041-44e5-8df0-c8c2420dfff2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nbrcn"
Oct 30 00:46:55.945224 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:55.945186    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/145c8ccc-6041-44e5-8df0-c8c2420dfff2-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-nbrcn\" (UID: \"145c8ccc-6041-44e5-8df0-c8c2420dfff2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nbrcn"
Oct 30 00:46:55.945385 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:55.945236    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npbcg\" (UniqueName: \"kubernetes.io/projected/145c8ccc-6041-44e5-8df0-c8c2420dfff2-kube-api-access-npbcg\") pod \"kserve-module-controller-manager-7566cf5fc8-nbrcn\" (UID: \"145c8ccc-6041-44e5-8df0-c8c2420dfff2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nbrcn"
Oct 30 00:46:55.945385 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:55.945272    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/145c8ccc-6041-44e5-8df0-c8c2420dfff2-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-nbrcn\" (UID: \"145c8ccc-6041-44e5-8df0-c8c2420dfff2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nbrcn"
Oct 30 00:46:55.945385 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:55.945293    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/145c8ccc-6041-44e5-8df0-c8c2420dfff2-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-nbrcn\" (UID: \"145c8ccc-6041-44e5-8df0-c8c2420dfff2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nbrcn"
Oct 30 00:46:55.945579 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:55.945559    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/145c8ccc-6041-44e5-8df0-c8c2420dfff2-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-nbrcn\" (UID: \"145c8ccc-6041-44e5-8df0-c8c2420dfff2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nbrcn"
Oct 30 00:46:55.945712 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:55.945690    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/145c8ccc-6041-44e5-8df0-c8c2420dfff2-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-nbrcn\" (UID: \"145c8ccc-6041-44e5-8df0-c8c2420dfff2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nbrcn"
Oct 30 00:46:55.946233 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:55.946204    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/145c8ccc-6041-44e5-8df0-c8c2420dfff2-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-nbrcn\" (UID: \"145c8ccc-6041-44e5-8df0-c8c2420dfff2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nbrcn"
Oct 30 00:46:55.967519 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:55.967489    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npbcg\" (UniqueName: \"kubernetes.io/projected/145c8ccc-6041-44e5-8df0-c8c2420dfff2-kube-api-access-npbcg\") pod \"kserve-module-controller-manager-7566cf5fc8-nbrcn\" (UID: \"145c8ccc-6041-44e5-8df0-c8c2420dfff2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nbrcn"
Oct 30 00:46:56.045197 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:56.045110    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nbrcn"
Oct 30 00:46:56.353779 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:56.353696    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-nbrcn"]
Oct 30 00:46:56.358795 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:46:56.358757    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145c8ccc_6041_44e5_8df0_c8c2420dfff2.slice/crio-08a8f429da601165f1072e3f96c7cb0339ef4490c50d213f93c07631b366e015 WatchSource:0}: Error finding container 08a8f429da601165f1072e3f96c7cb0339ef4490c50d213f93c07631b366e015: Status 404 returned error can't find the container with id 08a8f429da601165f1072e3f96c7cb0339ef4490c50d213f93c07631b366e015
Oct 30 00:46:57.161672 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:57.161622    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nbrcn" event={"ID":"145c8ccc-6041-44e5-8df0-c8c2420dfff2","Type":"ContainerStarted","Data":"08a8f429da601165f1072e3f96c7cb0339ef4490c50d213f93c07631b366e015"}
Oct 30 00:46:59.171037 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:59.170995    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nbrcn" event={"ID":"145c8ccc-6041-44e5-8df0-c8c2420dfff2","Type":"ContainerStarted","Data":"45d4e0826df397ca581e9979329661bd6a04fe5b0a85543a2dc79b014450b626"}
Oct 30 00:46:59.171435 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:59.171136    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nbrcn"
Oct 30 00:46:59.221381 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:46:59.221331    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nbrcn" podStartSLOduration=2.38469373 podStartE2EDuration="4.22131759s" totalImagesPullingTime="1.83662386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:59.220482992 +0000 UTC m=+331.558584914" watchObservedRunningTime="2026-10-30 00:46:59.22131759 +0000 UTC m=+331.559419540"
Oct 30 00:47:19.038044 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:47:19.038002    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:47:20.176692 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:20.176654    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nbrcn"
Oct 30 00:47:22.553678 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:22.553641    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6cb9c66845-tcgp7"]
Oct 30 00:47:22.557082 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:22.557050    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6cb9c66845-tcgp7"
Oct 30 00:47:22.580184 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:22.580156    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:47:22.580421 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:22.580173    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 00:47:22.580421 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:22.580204    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-ntjzd\""
Oct 30 00:47:22.580421 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:22.580395    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 00:47:22.580763 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:22.580499    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 00:47:22.580763 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:22.580556    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 00:47:22.582124 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:22.582102    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6cb9c66845-tcgp7"]
Oct 30 00:47:22.652660 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:22.652624    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a634f651-82f4-4276-9405-9a1e9bad48e3-cert\") pod \"jobset-controller-manager-6cb9c66845-tcgp7\" (UID: \"a634f651-82f4-4276-9405-9a1e9bad48e3\") " pod="openshift-jobset-operator/jobset-controller-manager-6cb9c66845-tcgp7"
Oct 30 00:47:22.652845 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:22.652690    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a634f651-82f4-4276-9405-9a1e9bad48e3-metrics-certs\") pod \"jobset-controller-manager-6cb9c66845-tcgp7\" (UID: \"a634f651-82f4-4276-9405-9a1e9bad48e3\") " pod="openshift-jobset-operator/jobset-controller-manager-6cb9c66845-tcgp7"
Oct 30 00:47:22.652845 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:22.652747    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvr58\" (UniqueName: \"kubernetes.io/projected/a634f651-82f4-4276-9405-9a1e9bad48e3-kube-api-access-pvr58\") pod \"jobset-controller-manager-6cb9c66845-tcgp7\" (UID: \"a634f651-82f4-4276-9405-9a1e9bad48e3\") " pod="openshift-jobset-operator/jobset-controller-manager-6cb9c66845-tcgp7"
Oct 30 00:47:22.652923 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:22.652855    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a634f651-82f4-4276-9405-9a1e9bad48e3-manager-config\") pod \"jobset-controller-manager-6cb9c66845-tcgp7\" (UID: \"a634f651-82f4-4276-9405-9a1e9bad48e3\") " pod="openshift-jobset-operator/jobset-controller-manager-6cb9c66845-tcgp7"
Oct 30 00:47:22.753985 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:22.753946    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a634f651-82f4-4276-9405-9a1e9bad48e3-manager-config\") pod \"jobset-controller-manager-6cb9c66845-tcgp7\" (UID: \"a634f651-82f4-4276-9405-9a1e9bad48e3\") " pod="openshift-jobset-operator/jobset-controller-manager-6cb9c66845-tcgp7"
Oct 30 00:47:22.754207 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:22.754005    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a634f651-82f4-4276-9405-9a1e9bad48e3-cert\") pod \"jobset-controller-manager-6cb9c66845-tcgp7\" (UID: \"a634f651-82f4-4276-9405-9a1e9bad48e3\") " pod="openshift-jobset-operator/jobset-controller-manager-6cb9c66845-tcgp7"
Oct 30 00:47:22.754207 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:22.754039    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a634f651-82f4-4276-9405-9a1e9bad48e3-metrics-certs\") pod \"jobset-controller-manager-6cb9c66845-tcgp7\" (UID: \"a634f651-82f4-4276-9405-9a1e9bad48e3\") " pod="openshift-jobset-operator/jobset-controller-manager-6cb9c66845-tcgp7"
Oct 30 00:47:22.754207 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:22.754059    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvr58\" (UniqueName: \"kubernetes.io/projected/a634f651-82f4-4276-9405-9a1e9bad48e3-kube-api-access-pvr58\") pod \"jobset-controller-manager-6cb9c66845-tcgp7\" (UID: \"a634f651-82f4-4276-9405-9a1e9bad48e3\") " pod="openshift-jobset-operator/jobset-controller-manager-6cb9c66845-tcgp7"
Oct 30 00:47:22.754706 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:22.754680    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a634f651-82f4-4276-9405-9a1e9bad48e3-manager-config\") pod \"jobset-controller-manager-6cb9c66845-tcgp7\" (UID: \"a634f651-82f4-4276-9405-9a1e9bad48e3\") " pod="openshift-jobset-operator/jobset-controller-manager-6cb9c66845-tcgp7"
Oct 30 00:47:22.756566 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:22.756542    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a634f651-82f4-4276-9405-9a1e9bad48e3-metrics-certs\") pod \"jobset-controller-manager-6cb9c66845-tcgp7\" (UID: \"a634f651-82f4-4276-9405-9a1e9bad48e3\") " pod="openshift-jobset-operator/jobset-controller-manager-6cb9c66845-tcgp7"
Oct 30 00:47:22.756777 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:22.756757    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a634f651-82f4-4276-9405-9a1e9bad48e3-cert\") pod \"jobset-controller-manager-6cb9c66845-tcgp7\" (UID: \"a634f651-82f4-4276-9405-9a1e9bad48e3\") " pod="openshift-jobset-operator/jobset-controller-manager-6cb9c66845-tcgp7"
Oct 30 00:47:22.776418 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:22.776384    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvr58\" (UniqueName: \"kubernetes.io/projected/a634f651-82f4-4276-9405-9a1e9bad48e3-kube-api-access-pvr58\") pod \"jobset-controller-manager-6cb9c66845-tcgp7\" (UID: \"a634f651-82f4-4276-9405-9a1e9bad48e3\") " pod="openshift-jobset-operator/jobset-controller-manager-6cb9c66845-tcgp7"
Oct 30 00:47:22.869442 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:22.869363    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6cb9c66845-tcgp7"
Oct 30 00:47:23.162585 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:23.162512    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6cb9c66845-tcgp7"]
Oct 30 00:47:23.167284 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:47:23.167248    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda634f651_82f4_4276_9405_9a1e9bad48e3.slice/crio-f773f8d7da8a47e407211f1aab0bcd2a2c547ebd191644cdafaf5fd5ed4f544f WatchSource:0}: Error finding container f773f8d7da8a47e407211f1aab0bcd2a2c547ebd191644cdafaf5fd5ed4f544f: Status 404 returned error can't find the container with id f773f8d7da8a47e407211f1aab0bcd2a2c547ebd191644cdafaf5fd5ed4f544f
Oct 30 00:47:23.268043 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:23.268003    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6cb9c66845-tcgp7" event={"ID":"a634f651-82f4-4276-9405-9a1e9bad48e3","Type":"ContainerStarted","Data":"f773f8d7da8a47e407211f1aab0bcd2a2c547ebd191644cdafaf5fd5ed4f544f"}
Oct 30 00:47:27.287481 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:27.287437    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6cb9c66845-tcgp7" event={"ID":"a634f651-82f4-4276-9405-9a1e9bad48e3","Type":"ContainerStarted","Data":"3f916dfe769ee6420dd4306f8b707a5fd9ecf07525a003bce52e9778e539ddc3"}
Oct 30 00:47:27.287957 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:27.287547    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6cb9c66845-tcgp7"
Oct 30 00:47:27.325850 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:27.325800    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-6cb9c66845-tcgp7" podStartSLOduration=1.8220599210000001 podStartE2EDuration="5.325786856s" totalImagesPullingTime="3.503726935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:27.324345338 +0000 UTC m=+359.662447249" watchObservedRunningTime="2026-10-30 00:47:27.325786856 +0000 UTC m=+359.663888766"
Oct 30 00:47:38.297742 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:47:38.297710    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6cb9c66845-tcgp7"
Oct 30 00:47:49.038822 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:47:49.038781    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:48:00.254994 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.254958    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"]
Oct 30 00:48:00.255529 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.255173    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt" podUID="e9651f3a-0816-4f3a-8792-2951ba4bc28a" containerName="discovery" containerID="cri-o://8065bc590afe6b0336ba260dbdf3f758336319aa10d2d04b5ae58f33228a741d" gracePeriod=30
Oct 30 00:48:00.427035 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.427006    4864 generic.go:356] "Generic (PLEG): container finished" podID="e9651f3a-0816-4f3a-8792-2951ba4bc28a" containerID="8065bc590afe6b0336ba260dbdf3f758336319aa10d2d04b5ae58f33228a741d" exitCode=0
Oct 30 00:48:00.427244 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.427212    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt" event={"ID":"e9651f3a-0816-4f3a-8792-2951ba4bc28a","Type":"ContainerDied","Data":"8065bc590afe6b0336ba260dbdf3f758336319aa10d2d04b5ae58f33228a741d"}
Oct 30 00:48:00.497372 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.497349    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:48:00.582928 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.582853    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-kubeconfig\") pod \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") "
Oct 30 00:48:00.582928 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.582908    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-csr-ca-configmap\") pod \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") "
Oct 30 00:48:00.583162 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.582978    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-token\" (UniqueName: \"kubernetes.io/projected/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-token\") pod \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") "
Oct 30 00:48:00.583162 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.583007    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9651f3a-0816-4f3a-8792-2951ba4bc28a-kube-api-access-8m5v2\" (UniqueName: \"kubernetes.io/projected/e9651f3a-0816-4f3a-8792-2951ba4bc28a-kube-api-access-8m5v2\") pod \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") "
Oct 30 00:48:00.583162 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.583059    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9651f3a-0816-4f3a-8792-2951ba4bc28a-cacerts\" (UniqueName: \"kubernetes.io/secret/e9651f3a-0816-4f3a-8792-2951ba4bc28a-cacerts\") pod \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") "
Oct 30 00:48:00.583162 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.583131    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-csr-dns-cert\") pod \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") "
Oct 30 00:48:00.583380 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.583166    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9651f3a-0816-4f3a-8792-2951ba4bc28a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9651f3a-0816-4f3a-8792-2951ba4bc28a-local-certs\") pod \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\" (UID: \"e9651f3a-0816-4f3a-8792-2951ba4bc28a\") "
Oct 30 00:48:00.583504 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.583472    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-csr-ca-configmap" pod "e9651f3a-0816-4f3a-8792-2951ba4bc28a" (UID: "e9651f3a-0816-4f3a-8792-2951ba4bc28a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:48:00.585419 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.585390    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-token" pod "e9651f3a-0816-4f3a-8792-2951ba4bc28a" (UID: "e9651f3a-0816-4f3a-8792-2951ba4bc28a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:48:00.585419 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.585404    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9651f3a-0816-4f3a-8792-2951ba4bc28a-kube-api-access-8m5v2" pod "e9651f3a-0816-4f3a-8792-2951ba4bc28a" (UID: "e9651f3a-0816-4f3a-8792-2951ba4bc28a"). InnerVolumeSpecName "kube-api-access-8m5v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:48:00.585614 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.585595    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9651f3a-0816-4f3a-8792-2951ba4bc28a-cacerts" pod "e9651f3a-0816-4f3a-8792-2951ba4bc28a" (UID: "e9651f3a-0816-4f3a-8792-2951ba4bc28a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:48:00.585740 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.585718    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-kubeconfig" pod "e9651f3a-0816-4f3a-8792-2951ba4bc28a" (UID: "e9651f3a-0816-4f3a-8792-2951ba4bc28a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:48:00.585830 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.585810    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9651f3a-0816-4f3a-8792-2951ba4bc28a-local-certs" pod "e9651f3a-0816-4f3a-8792-2951ba4bc28a" (UID: "e9651f3a-0816-4f3a-8792-2951ba4bc28a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:48:00.585885 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.585822    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-csr-dns-cert" pod "e9651f3a-0816-4f3a-8792-2951ba4bc28a" (UID: "e9651f3a-0816-4f3a-8792-2951ba4bc28a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:48:00.684164 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.684119    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:48:00.684164 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.684159    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m5v2\" (UniqueName: \"kubernetes.io/projected/e9651f3a-0816-4f3a-8792-2951ba4bc28a-kube-api-access-8m5v2\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:48:00.684164 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.684173    4864 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9651f3a-0816-4f3a-8792-2951ba4bc28a-cacerts\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:48:00.684382 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.684187    4864 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-csr-dns-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:48:00.684382 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.684199    4864 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9651f3a-0816-4f3a-8792-2951ba4bc28a-local-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:48:00.684382 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.684211    4864 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-kubeconfig\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:48:00.684382 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:00.684222    4864 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9651f3a-0816-4f3a-8792-2951ba4bc28a-istio-csr-ca-configmap\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:48:01.432702 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.432673    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"
Oct 30 00:48:01.432702 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.432674    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt" event={"ID":"e9651f3a-0816-4f3a-8792-2951ba4bc28a","Type":"ContainerDied","Data":"6460b790ab339d32924e792bb3dc3df42d39de41ae1d352df63d65889d1f8ac2"}
Oct 30 00:48:01.433223 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.432744    4864 scope.go:122] "RemoveContainer" containerID="8065bc590afe6b0336ba260dbdf3f758336319aa10d2d04b5ae58f33228a741d"
Oct 30 00:48:01.491719 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.491687    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"]
Oct 30 00:48:01.501734 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.501707    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-c7wlt"]
Oct 30 00:48:01.719083 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.719050    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tlc9c"]
Oct 30 00:48:01.719392 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.719372    4864 cpu_manager.go:414] "containerMap: removing container" podUID="e9651f3a-0816-4f3a-8792-2951ba4bc28a" containerName="discovery"
Oct 30 00:48:01.719392 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.719388    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9651f3a-0816-4f3a-8792-2951ba4bc28a" containerName="discovery"
Oct 30 00:48:01.719575 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.719477    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9651f3a-0816-4f3a-8792-2951ba4bc28a" containerName="discovery"
Oct 30 00:48:01.725269 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.725247    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tlc9c"
Oct 30 00:48:01.744685 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.744640    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 00:48:01.745008 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.744980    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 00:48:01.746341 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.746314    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tlc9c"]
Oct 30 00:48:01.791715 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.791685    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9651f3a-0816-4f3a-8792-2951ba4bc28a" path="/var/lib/kubelet/pods/e9651f3a-0816-4f3a-8792-2951ba4bc28a/volumes"
Oct 30 00:48:01.802454 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.802432    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 00:48:01.802577 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.802523    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-97qzf\""
Oct 30 00:48:01.893344 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.893313    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/62dcd375-7d44-4774-b18f-fb601a98b643-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tlc9c\" (UID: \"62dcd375-7d44-4774-b18f-fb601a98b643\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tlc9c"
Oct 30 00:48:01.893344 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.893345    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62dcd375-7d44-4774-b18f-fb601a98b643-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tlc9c\" (UID: \"62dcd375-7d44-4774-b18f-fb601a98b643\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tlc9c"
Oct 30 00:48:01.893597 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.893365    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b8h6\" (UniqueName: \"kubernetes.io/projected/62dcd375-7d44-4774-b18f-fb601a98b643-kube-api-access-2b8h6\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tlc9c\" (UID: \"62dcd375-7d44-4774-b18f-fb601a98b643\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tlc9c"
Oct 30 00:48:01.893664 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.893635    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/62dcd375-7d44-4774-b18f-fb601a98b643-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tlc9c\" (UID: \"62dcd375-7d44-4774-b18f-fb601a98b643\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tlc9c"
Oct 30 00:48:01.994140 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.994047    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/62dcd375-7d44-4774-b18f-fb601a98b643-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tlc9c\" (UID: \"62dcd375-7d44-4774-b18f-fb601a98b643\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tlc9c"
Oct 30 00:48:01.994140 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.994125    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/62dcd375-7d44-4774-b18f-fb601a98b643-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tlc9c\" (UID: \"62dcd375-7d44-4774-b18f-fb601a98b643\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tlc9c"
Oct 30 00:48:01.994374 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.994150    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62dcd375-7d44-4774-b18f-fb601a98b643-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tlc9c\" (UID: \"62dcd375-7d44-4774-b18f-fb601a98b643\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tlc9c"
Oct 30 00:48:01.994374 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.994175    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b8h6\" (UniqueName: \"kubernetes.io/projected/62dcd375-7d44-4774-b18f-fb601a98b643-kube-api-access-2b8h6\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tlc9c\" (UID: \"62dcd375-7d44-4774-b18f-fb601a98b643\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tlc9c"
Oct 30 00:48:01.994878 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.994855    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62dcd375-7d44-4774-b18f-fb601a98b643-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tlc9c\" (UID: \"62dcd375-7d44-4774-b18f-fb601a98b643\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tlc9c"
Oct 30 00:48:01.995129 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.995108    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/62dcd375-7d44-4774-b18f-fb601a98b643-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tlc9c\" (UID: \"62dcd375-7d44-4774-b18f-fb601a98b643\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tlc9c"
Oct 30 00:48:01.996586 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:01.996559    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/62dcd375-7d44-4774-b18f-fb601a98b643-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tlc9c\" (UID: \"62dcd375-7d44-4774-b18f-fb601a98b643\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tlc9c"
Oct 30 00:48:02.015815 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:02.015789    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b8h6\" (UniqueName: \"kubernetes.io/projected/62dcd375-7d44-4774-b18f-fb601a98b643-kube-api-access-2b8h6\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tlc9c\" (UID: \"62dcd375-7d44-4774-b18f-fb601a98b643\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tlc9c"
Oct 30 00:48:02.038783 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:02.038757    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tlc9c"
Oct 30 00:48:02.332376 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:02.332299    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tlc9c"]
Oct 30 00:48:02.337258 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:48:02.337230    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62dcd375_7d44_4774_b18f_fb601a98b643.slice/crio-ff78f39e13319b81ae504b89c96b0f3adbd94171f74993b592cd58e64b6019cb WatchSource:0}: Error finding container ff78f39e13319b81ae504b89c96b0f3adbd94171f74993b592cd58e64b6019cb: Status 404 returned error can't find the container with id ff78f39e13319b81ae504b89c96b0f3adbd94171f74993b592cd58e64b6019cb
Oct 30 00:48:02.439125 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:02.439092    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tlc9c" event={"ID":"62dcd375-7d44-4774-b18f-fb601a98b643","Type":"ContainerStarted","Data":"ff78f39e13319b81ae504b89c96b0f3adbd94171f74993b592cd58e64b6019cb"}
Oct 30 00:48:19.039983 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:48:19.039908    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:48:19.521220 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:19.521180    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tlc9c" event={"ID":"62dcd375-7d44-4774-b18f-fb601a98b643","Type":"ContainerStarted","Data":"24736a303a8f45e6cc72006e34635afa46447f08b8d83fab94700358f3378fd8"}
Oct 30 00:48:19.521407 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:19.521281    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tlc9c"
Oct 30 00:48:19.559851 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:19.559791    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tlc9c" podStartSLOduration=1.766376507 podStartE2EDuration="18.559776455s" totalImagesPullingTime="16.793399948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:19.558514774 +0000 UTC m=+411.896616698" watchObservedRunningTime="2026-10-30 00:48:19.559776455 +0000 UTC m=+411.897878364"
Oct 30 00:48:30.531456 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:30.531424    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tlc9c"
Oct 30 00:48:32.273509 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:32.273478    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn"]
Oct 30 00:48:32.278264 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:32.278242    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn"
Oct 30 00:48:32.301376 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:32.301342    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:48:32.301548 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:32.301530    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 00:48:32.301614 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:32.301603    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 00:48:32.301890 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:32.301864    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:48:32.303305 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:32.303282    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn"]
Oct 30 00:48:32.303400 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:32.303345    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p8xs2\""
Oct 30 00:48:32.347618 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:32.347583    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b69c6934-beaa-4192-b3b3-0dd7535e4226-maas-api-tls\") pod \"maas-api-5d98c8675d-cxndn\" (UID: \"b69c6934-beaa-4192-b3b3-0dd7535e4226\") " pod="odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn"
Oct 30 00:48:32.347758 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:32.347638    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b69c6934-beaa-4192-b3b3-0dd7535e4226-openshift-service-ca\") pod \"maas-api-5d98c8675d-cxndn\" (UID: \"b69c6934-beaa-4192-b3b3-0dd7535e4226\") " pod="odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn"
Oct 30 00:48:32.347758 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:32.347661    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b69c6934-beaa-4192-b3b3-0dd7535e4226-metrics-certs\") pod \"maas-api-5d98c8675d-cxndn\" (UID: \"b69c6934-beaa-4192-b3b3-0dd7535e4226\") " pod="odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn"
Oct 30 00:48:32.347845 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:32.347753    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvd62\" (UniqueName: \"kubernetes.io/projected/b69c6934-beaa-4192-b3b3-0dd7535e4226-kube-api-access-gvd62\") pod \"maas-api-5d98c8675d-cxndn\" (UID: \"b69c6934-beaa-4192-b3b3-0dd7535e4226\") " pod="odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn"
Oct 30 00:48:32.448849 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:32.448819    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b69c6934-beaa-4192-b3b3-0dd7535e4226-maas-api-tls\") pod \"maas-api-5d98c8675d-cxndn\" (UID: \"b69c6934-beaa-4192-b3b3-0dd7535e4226\") " pod="odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn"
Oct 30 00:48:32.449040 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:32.448863    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b69c6934-beaa-4192-b3b3-0dd7535e4226-openshift-service-ca\") pod \"maas-api-5d98c8675d-cxndn\" (UID: \"b69c6934-beaa-4192-b3b3-0dd7535e4226\") " pod="odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn"
Oct 30 00:48:32.449040 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:32.448883    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b69c6934-beaa-4192-b3b3-0dd7535e4226-metrics-certs\") pod \"maas-api-5d98c8675d-cxndn\" (UID: \"b69c6934-beaa-4192-b3b3-0dd7535e4226\") " pod="odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn"
Oct 30 00:48:32.449040 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:32.448916    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvd62\" (UniqueName: \"kubernetes.io/projected/b69c6934-beaa-4192-b3b3-0dd7535e4226-kube-api-access-gvd62\") pod \"maas-api-5d98c8675d-cxndn\" (UID: \"b69c6934-beaa-4192-b3b3-0dd7535e4226\") " pod="odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn"
Oct 30 00:48:32.449525 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:32.449497    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b69c6934-beaa-4192-b3b3-0dd7535e4226-openshift-service-ca\") pod \"maas-api-5d98c8675d-cxndn\" (UID: \"b69c6934-beaa-4192-b3b3-0dd7535e4226\") " pod="odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn"
Oct 30 00:48:32.451364 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:32.451337    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b69c6934-beaa-4192-b3b3-0dd7535e4226-maas-api-tls\") pod \"maas-api-5d98c8675d-cxndn\" (UID: \"b69c6934-beaa-4192-b3b3-0dd7535e4226\") " pod="odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn"
Oct 30 00:48:32.451463 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:32.451371    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b69c6934-beaa-4192-b3b3-0dd7535e4226-metrics-certs\") pod \"maas-api-5d98c8675d-cxndn\" (UID: \"b69c6934-beaa-4192-b3b3-0dd7535e4226\") " pod="odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn"
Oct 30 00:48:32.470032 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:32.470011    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvd62\" (UniqueName: \"kubernetes.io/projected/b69c6934-beaa-4192-b3b3-0dd7535e4226-kube-api-access-gvd62\") pod \"maas-api-5d98c8675d-cxndn\" (UID: \"b69c6934-beaa-4192-b3b3-0dd7535e4226\") " pod="odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn"
Oct 30 00:48:32.590731 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:32.590649    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn"
Oct 30 00:48:32.889465 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:32.889390    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn"]
Oct 30 00:48:32.895049 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:48:32.895018    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69c6934_beaa_4192_b3b3_0dd7535e4226.slice/crio-84b70f9ef73bfa18be6232c040c2fd4c24a545555dda2692b66ea2a598095801 WatchSource:0}: Error finding container 84b70f9ef73bfa18be6232c040c2fd4c24a545555dda2692b66ea2a598095801: Status 404 returned error can't find the container with id 84b70f9ef73bfa18be6232c040c2fd4c24a545555dda2692b66ea2a598095801
Oct 30 00:48:33.577748 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:33.577704    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn" event={"ID":"b69c6934-beaa-4192-b3b3-0dd7535e4226","Type":"ContainerStarted","Data":"84b70f9ef73bfa18be6232c040c2fd4c24a545555dda2692b66ea2a598095801"}
Oct 30 00:48:35.589235 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:35.589141    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn" event={"ID":"b69c6934-beaa-4192-b3b3-0dd7535e4226","Type":"ContainerStarted","Data":"3a99df7995441c3cb84b69ac2a55d7f81f4df1a4115ae8e41954cbd28e854ee0"}
Oct 30 00:48:35.589615 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:35.589252    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn"
Oct 30 00:48:35.628503 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:35.628443    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn" podStartSLOduration=1.239266414 podStartE2EDuration="3.628424269s" totalImagesPullingTime="2.389157855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:35.627595037 +0000 UTC m=+427.965696947" watchObservedRunningTime="2026-10-30 00:48:35.628424269 +0000 UTC m=+427.966526181"
Oct 30 00:48:41.598987 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:41.598961    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn"
Oct 30 00:48:42.529743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:42.529707    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d5c8bc49-t2rj5"]
Oct 30 00:48:42.533010 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:42.532988    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d5c8bc49-t2rj5"
Oct 30 00:48:42.557589 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:42.557563    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d5c8bc49-t2rj5"]
Oct 30 00:48:42.641137 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:42.641102    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bc51d8f-34a6-4134-9033-90fb1c5cabc5-maas-api-tls\") pod \"maas-api-6d5c8bc49-t2rj5\" (UID: \"8bc51d8f-34a6-4134-9033-90fb1c5cabc5\") " pod="odh-ai-gateway-infra/maas-api-6d5c8bc49-t2rj5"
Oct 30 00:48:42.641548 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:42.641150    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbfdc\" (UniqueName: \"kubernetes.io/projected/8bc51d8f-34a6-4134-9033-90fb1c5cabc5-kube-api-access-pbfdc\") pod \"maas-api-6d5c8bc49-t2rj5\" (UID: \"8bc51d8f-34a6-4134-9033-90fb1c5cabc5\") " pod="odh-ai-gateway-infra/maas-api-6d5c8bc49-t2rj5"
Oct 30 00:48:42.641548 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:42.641247    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bc51d8f-34a6-4134-9033-90fb1c5cabc5-metrics-certs\") pod \"maas-api-6d5c8bc49-t2rj5\" (UID: \"8bc51d8f-34a6-4134-9033-90fb1c5cabc5\") " pod="odh-ai-gateway-infra/maas-api-6d5c8bc49-t2rj5"
Oct 30 00:48:42.641548 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:42.641280    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bc51d8f-34a6-4134-9033-90fb1c5cabc5-openshift-service-ca\") pod \"maas-api-6d5c8bc49-t2rj5\" (UID: \"8bc51d8f-34a6-4134-9033-90fb1c5cabc5\") " pod="odh-ai-gateway-infra/maas-api-6d5c8bc49-t2rj5"
Oct 30 00:48:42.742310 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:42.742272    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bc51d8f-34a6-4134-9033-90fb1c5cabc5-maas-api-tls\") pod \"maas-api-6d5c8bc49-t2rj5\" (UID: \"8bc51d8f-34a6-4134-9033-90fb1c5cabc5\") " pod="odh-ai-gateway-infra/maas-api-6d5c8bc49-t2rj5"
Oct 30 00:48:42.742310 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:42.742317    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbfdc\" (UniqueName: \"kubernetes.io/projected/8bc51d8f-34a6-4134-9033-90fb1c5cabc5-kube-api-access-pbfdc\") pod \"maas-api-6d5c8bc49-t2rj5\" (UID: \"8bc51d8f-34a6-4134-9033-90fb1c5cabc5\") " pod="odh-ai-gateway-infra/maas-api-6d5c8bc49-t2rj5"
Oct 30 00:48:42.742559 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:42.742416    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bc51d8f-34a6-4134-9033-90fb1c5cabc5-metrics-certs\") pod \"maas-api-6d5c8bc49-t2rj5\" (UID: \"8bc51d8f-34a6-4134-9033-90fb1c5cabc5\") " pod="odh-ai-gateway-infra/maas-api-6d5c8bc49-t2rj5"
Oct 30 00:48:42.742559 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:42.742439    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bc51d8f-34a6-4134-9033-90fb1c5cabc5-openshift-service-ca\") pod \"maas-api-6d5c8bc49-t2rj5\" (UID: \"8bc51d8f-34a6-4134-9033-90fb1c5cabc5\") " pod="odh-ai-gateway-infra/maas-api-6d5c8bc49-t2rj5"
Oct 30 00:48:42.743121 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:42.743096    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bc51d8f-34a6-4134-9033-90fb1c5cabc5-openshift-service-ca\") pod \"maas-api-6d5c8bc49-t2rj5\" (UID: \"8bc51d8f-34a6-4134-9033-90fb1c5cabc5\") " pod="odh-ai-gateway-infra/maas-api-6d5c8bc49-t2rj5"
Oct 30 00:48:42.744863 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:42.744833    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bc51d8f-34a6-4134-9033-90fb1c5cabc5-metrics-certs\") pod \"maas-api-6d5c8bc49-t2rj5\" (UID: \"8bc51d8f-34a6-4134-9033-90fb1c5cabc5\") " pod="odh-ai-gateway-infra/maas-api-6d5c8bc49-t2rj5"
Oct 30 00:48:42.744958 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:42.744842    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bc51d8f-34a6-4134-9033-90fb1c5cabc5-maas-api-tls\") pod \"maas-api-6d5c8bc49-t2rj5\" (UID: \"8bc51d8f-34a6-4134-9033-90fb1c5cabc5\") " pod="odh-ai-gateway-infra/maas-api-6d5c8bc49-t2rj5"
Oct 30 00:48:42.764015 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:42.763992    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbfdc\" (UniqueName: \"kubernetes.io/projected/8bc51d8f-34a6-4134-9033-90fb1c5cabc5-kube-api-access-pbfdc\") pod \"maas-api-6d5c8bc49-t2rj5\" (UID: \"8bc51d8f-34a6-4134-9033-90fb1c5cabc5\") " pod="odh-ai-gateway-infra/maas-api-6d5c8bc49-t2rj5"
Oct 30 00:48:42.845569 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:42.845500    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d5c8bc49-t2rj5"
Oct 30 00:48:43.143887 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:43.143802    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d5c8bc49-t2rj5"]
Oct 30 00:48:43.148705 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:48:43.148671    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc51d8f_34a6_4134_9033_90fb1c5cabc5.slice/crio-58867e1ae590dca2ed765bdb4d9dbbd5acacd1906e04e19137b6690c86f7cfc5 WatchSource:0}: Error finding container 58867e1ae590dca2ed765bdb4d9dbbd5acacd1906e04e19137b6690c86f7cfc5: Status 404 returned error can't find the container with id 58867e1ae590dca2ed765bdb4d9dbbd5acacd1906e04e19137b6690c86f7cfc5
Oct 30 00:48:43.622563 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:43.622527    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d5c8bc49-t2rj5" event={"ID":"8bc51d8f-34a6-4134-9033-90fb1c5cabc5","Type":"ContainerStarted","Data":"d4996752e6ba5ba413f3d8787533ea43d2e930bb8838653dce000e379ed257e7"}
Oct 30 00:48:43.622729 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:43.622579    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d5c8bc49-t2rj5" event={"ID":"8bc51d8f-34a6-4134-9033-90fb1c5cabc5","Type":"ContainerStarted","Data":"58867e1ae590dca2ed765bdb4d9dbbd5acacd1906e04e19137b6690c86f7cfc5"}
Oct 30 00:48:43.622729 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:43.622644    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d5c8bc49-t2rj5"
Oct 30 00:48:43.662313 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:43.662262    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d5c8bc49-t2rj5" podStartSLOduration=1.369161997 podStartE2EDuration="1.662243219s" totalImagesPullingTime="293.081222ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:43.661571008 +0000 UTC m=+435.999672917" watchObservedRunningTime="2026-10-30 00:48:43.662243219 +0000 UTC m=+436.000345129"
Oct 30 00:48:49.039361 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:48:49.039318    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:48:49.632290 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:49.632259    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d5c8bc49-t2rj5"
Oct 30 00:48:49.704193 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:49.704162    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn"]
Oct 30 00:48:49.704390 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:49.704349    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn" podUID="b69c6934-beaa-4192-b3b3-0dd7535e4226" containerName="maas-api" containerID="cri-o://3a99df7995441c3cb84b69ac2a55d7f81f4df1a4115ae8e41954cbd28e854ee0" gracePeriod=30
Oct 30 00:48:50.491344 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:50.491315    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn"
Oct 30 00:48:50.615345 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:50.615254    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b69c6934-beaa-4192-b3b3-0dd7535e4226-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b69c6934-beaa-4192-b3b3-0dd7535e4226-openshift-service-ca\") pod \"b69c6934-beaa-4192-b3b3-0dd7535e4226\" (UID: \"b69c6934-beaa-4192-b3b3-0dd7535e4226\") "
Oct 30 00:48:50.615345 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:50.615292    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b69c6934-beaa-4192-b3b3-0dd7535e4226-kube-api-access-gvd62\" (UniqueName: \"kubernetes.io/projected/b69c6934-beaa-4192-b3b3-0dd7535e4226-kube-api-access-gvd62\") pod \"b69c6934-beaa-4192-b3b3-0dd7535e4226\" (UID: \"b69c6934-beaa-4192-b3b3-0dd7535e4226\") "
Oct 30 00:48:50.615345 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:50.615315    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b69c6934-beaa-4192-b3b3-0dd7535e4226-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b69c6934-beaa-4192-b3b3-0dd7535e4226-metrics-certs\") pod \"b69c6934-beaa-4192-b3b3-0dd7535e4226\" (UID: \"b69c6934-beaa-4192-b3b3-0dd7535e4226\") "
Oct 30 00:48:50.615627 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:50.615384    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b69c6934-beaa-4192-b3b3-0dd7535e4226-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b69c6934-beaa-4192-b3b3-0dd7535e4226-maas-api-tls\") pod \"b69c6934-beaa-4192-b3b3-0dd7535e4226\" (UID: \"b69c6934-beaa-4192-b3b3-0dd7535e4226\") "
Oct 30 00:48:50.615717 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:50.615687    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b69c6934-beaa-4192-b3b3-0dd7535e4226-openshift-service-ca" pod "b69c6934-beaa-4192-b3b3-0dd7535e4226" (UID: "b69c6934-beaa-4192-b3b3-0dd7535e4226"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:48:50.617675 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:50.617644    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b69c6934-beaa-4192-b3b3-0dd7535e4226-metrics-certs" pod "b69c6934-beaa-4192-b3b3-0dd7535e4226" (UID: "b69c6934-beaa-4192-b3b3-0dd7535e4226"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:48:50.617800 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:50.617735    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69c6934-beaa-4192-b3b3-0dd7535e4226-kube-api-access-gvd62" pod "b69c6934-beaa-4192-b3b3-0dd7535e4226" (UID: "b69c6934-beaa-4192-b3b3-0dd7535e4226"). InnerVolumeSpecName "kube-api-access-gvd62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:48:50.617800 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:50.617752    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b69c6934-beaa-4192-b3b3-0dd7535e4226-maas-api-tls" pod "b69c6934-beaa-4192-b3b3-0dd7535e4226" (UID: "b69c6934-beaa-4192-b3b3-0dd7535e4226"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:48:50.652307 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:50.652275    4864 generic.go:356] "Generic (PLEG): container finished" podID="b69c6934-beaa-4192-b3b3-0dd7535e4226" containerID="3a99df7995441c3cb84b69ac2a55d7f81f4df1a4115ae8e41954cbd28e854ee0" exitCode=0
Oct 30 00:48:50.652476 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:50.652331    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn"
Oct 30 00:48:50.652476 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:50.652335    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn" event={"ID":"b69c6934-beaa-4192-b3b3-0dd7535e4226","Type":"ContainerDied","Data":"3a99df7995441c3cb84b69ac2a55d7f81f4df1a4115ae8e41954cbd28e854ee0"}
Oct 30 00:48:50.652476 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:50.652402    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn" event={"ID":"b69c6934-beaa-4192-b3b3-0dd7535e4226","Type":"ContainerDied","Data":"84b70f9ef73bfa18be6232c040c2fd4c24a545555dda2692b66ea2a598095801"}
Oct 30 00:48:50.652476 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:50.652423    4864 scope.go:122] "RemoveContainer" containerID="3a99df7995441c3cb84b69ac2a55d7f81f4df1a4115ae8e41954cbd28e854ee0"
Oct 30 00:48:50.661367 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:50.661351    4864 scope.go:122] "RemoveContainer" containerID="3a99df7995441c3cb84b69ac2a55d7f81f4df1a4115ae8e41954cbd28e854ee0"
Oct 30 00:48:50.661615 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:48:50.661593    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a99df7995441c3cb84b69ac2a55d7f81f4df1a4115ae8e41954cbd28e854ee0\": container with ID starting with 3a99df7995441c3cb84b69ac2a55d7f81f4df1a4115ae8e41954cbd28e854ee0 not found: ID does not exist" containerID="3a99df7995441c3cb84b69ac2a55d7f81f4df1a4115ae8e41954cbd28e854ee0"
Oct 30 00:48:50.661669 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:50.661625    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a99df7995441c3cb84b69ac2a55d7f81f4df1a4115ae8e41954cbd28e854ee0"} err="failed to get container status \"3a99df7995441c3cb84b69ac2a55d7f81f4df1a4115ae8e41954cbd28e854ee0\": rpc error: code = NotFound desc = could not find container \"3a99df7995441c3cb84b69ac2a55d7f81f4df1a4115ae8e41954cbd28e854ee0\": container with ID starting with 3a99df7995441c3cb84b69ac2a55d7f81f4df1a4115ae8e41954cbd28e854ee0 not found: ID does not exist"
Oct 30 00:48:50.697704 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:50.697673    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn"]
Oct 30 00:48:50.703020 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:50.702991    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d98c8675d-cxndn"]
Oct 30 00:48:50.715945 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:50.715925    4864 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b69c6934-beaa-4192-b3b3-0dd7535e4226-metrics-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:48:50.715945 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:50.715946    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b69c6934-beaa-4192-b3b3-0dd7535e4226-maas-api-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:48:50.716057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:50.715955    4864 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b69c6934-beaa-4192-b3b3-0dd7535e4226-openshift-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:48:50.716057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:50.715967    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvd62\" (UniqueName: \"kubernetes.io/projected/b69c6934-beaa-4192-b3b3-0dd7535e4226-kube-api-access-gvd62\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:48:51.790684 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:48:51.790651    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b69c6934-beaa-4192-b3b3-0dd7535e4226" path="/var/lib/kubelet/pods/b69c6934-beaa-4192-b3b3-0dd7535e4226/volumes"
Oct 30 00:49:19.038589 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:49:19.038553    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:49:19.716617 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:19.716577    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-44vqk"]
Oct 30 00:49:19.716958 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:19.716945    4864 cpu_manager.go:414] "containerMap: removing container" podUID="b69c6934-beaa-4192-b3b3-0dd7535e4226" containerName="maas-api"
Oct 30 00:49:19.717000 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:19.716958    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b69c6934-beaa-4192-b3b3-0dd7535e4226" containerName="maas-api"
Oct 30 00:49:19.717034 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:19.717016    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="b69c6934-beaa-4192-b3b3-0dd7535e4226" containerName="maas-api"
Oct 30 00:49:19.720031 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:19.720012    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-44vqk"
Oct 30 00:49:19.733293 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:19.733267    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 00:49:19.733293 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:19.733274    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 00:49:19.743671 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:19.743640    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-hqj84\""
Oct 30 00:49:19.745263 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:19.745239    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-44vqk"]
Oct 30 00:49:19.756135 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:19.756107    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b798b208-68eb-40ee-89b3-1a5f24654439-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-44vqk\" (UID: \"b798b208-68eb-40ee-89b3-1a5f24654439\") " pod="opendatahub/ai-gateway-controller-7d85675f98-44vqk"
Oct 30 00:49:19.756283 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:19.756257    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njvvt\" (UniqueName: \"kubernetes.io/projected/b798b208-68eb-40ee-89b3-1a5f24654439-kube-api-access-njvvt\") pod \"ai-gateway-controller-7d85675f98-44vqk\" (UID: \"b798b208-68eb-40ee-89b3-1a5f24654439\") " pod="opendatahub/ai-gateway-controller-7d85675f98-44vqk"
Oct 30 00:49:19.856990 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:19.856937    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b798b208-68eb-40ee-89b3-1a5f24654439-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-44vqk\" (UID: \"b798b208-68eb-40ee-89b3-1a5f24654439\") " pod="opendatahub/ai-gateway-controller-7d85675f98-44vqk"
Oct 30 00:49:19.857214 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:19.857016    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njvvt\" (UniqueName: \"kubernetes.io/projected/b798b208-68eb-40ee-89b3-1a5f24654439-kube-api-access-njvvt\") pod \"ai-gateway-controller-7d85675f98-44vqk\" (UID: \"b798b208-68eb-40ee-89b3-1a5f24654439\") " pod="opendatahub/ai-gateway-controller-7d85675f98-44vqk"
Oct 30 00:49:19.859523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:19.859498    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b798b208-68eb-40ee-89b3-1a5f24654439-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-44vqk\" (UID: \"b798b208-68eb-40ee-89b3-1a5f24654439\") " pod="opendatahub/ai-gateway-controller-7d85675f98-44vqk"
Oct 30 00:49:19.879221 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:19.879189    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njvvt\" (UniqueName: \"kubernetes.io/projected/b798b208-68eb-40ee-89b3-1a5f24654439-kube-api-access-njvvt\") pod \"ai-gateway-controller-7d85675f98-44vqk\" (UID: \"b798b208-68eb-40ee-89b3-1a5f24654439\") " pod="opendatahub/ai-gateway-controller-7d85675f98-44vqk"
Oct 30 00:49:20.032557 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:20.032468    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-44vqk"
Oct 30 00:49:20.326313 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:20.326237    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-44vqk"]
Oct 30 00:49:20.331404 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:49:20.331372    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb798b208_68eb_40ee_89b3_1a5f24654439.slice/crio-d2559fa859c15b9611d29f5f38dba01641d78e7b0465ed6a8f04d3517f3cb8ec WatchSource:0}: Error finding container d2559fa859c15b9611d29f5f38dba01641d78e7b0465ed6a8f04d3517f3cb8ec: Status 404 returned error can't find the container with id d2559fa859c15b9611d29f5f38dba01641d78e7b0465ed6a8f04d3517f3cb8ec
Oct 30 00:49:20.774704 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:20.774669    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-44vqk" event={"ID":"b798b208-68eb-40ee-89b3-1a5f24654439","Type":"ContainerStarted","Data":"d2559fa859c15b9611d29f5f38dba01641d78e7b0465ed6a8f04d3517f3cb8ec"}
Oct 30 00:49:22.784426 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:22.784385    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-44vqk" event={"ID":"b798b208-68eb-40ee-89b3-1a5f24654439","Type":"ContainerStarted","Data":"54b53e76654c643688c901c51b0ad7ae44bf11ad883f02146c4f2870ff42b3b4"}
Oct 30 00:49:22.784832 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:22.784464    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-44vqk"
Oct 30 00:49:22.823021 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:22.822972    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-44vqk" podStartSLOduration=2.384947157 podStartE2EDuration="3.822956454s" totalImagesPullingTime="1.438009297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:22.821787564 +0000 UTC m=+475.159889473" watchObservedRunningTime="2026-10-30 00:49:22.822956454 +0000 UTC m=+475.161058364"
Oct 30 00:49:24.140799 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.140758    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-w45z8"]
Oct 30 00:49:24.144476 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.144452    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w45z8"
Oct 30 00:49:24.157468 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.157452    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 00:49:24.168209 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.168182    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-m26xd\""
Oct 30 00:49:24.170051 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.170030    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-w45z8"]
Oct 30 00:49:24.191351 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.191325    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7l52\" (UniqueName: \"kubernetes.io/projected/a13e1eb5-b23d-49ac-9e1d-6b431e9f07fd-kube-api-access-p7l52\") pod \"payload-pre-processing-7b98bb678c-w45z8\" (UID: \"a13e1eb5-b23d-49ac-9e1d-6b431e9f07fd\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-w45z8"
Oct 30 00:49:24.191482 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.191465    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a13e1eb5-b23d-49ac-9e1d-6b431e9f07fd-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-w45z8\" (UID: \"a13e1eb5-b23d-49ac-9e1d-6b431e9f07fd\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-w45z8"
Oct 30 00:49:24.229427 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.229399    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-6k58k"]
Oct 30 00:49:24.232655 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.232637    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-6k58k"
Oct 30 00:49:24.252822 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.252792    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-6k58k"]
Oct 30 00:49:24.292579 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.292545    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqgwg\" (UniqueName: \"kubernetes.io/projected/d2a355c3-0703-422f-8ddb-555c4a364ade-kube-api-access-zqgwg\") pod \"payload-processing-5f78687d5b-6k58k\" (UID: \"d2a355c3-0703-422f-8ddb-555c4a364ade\") " pod="openshift-ingress/payload-processing-5f78687d5b-6k58k"
Oct 30 00:49:24.292752 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.292586    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7l52\" (UniqueName: \"kubernetes.io/projected/a13e1eb5-b23d-49ac-9e1d-6b431e9f07fd-kube-api-access-p7l52\") pod \"payload-pre-processing-7b98bb678c-w45z8\" (UID: \"a13e1eb5-b23d-49ac-9e1d-6b431e9f07fd\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-w45z8"
Oct 30 00:49:24.292752 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.292675    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a355c3-0703-422f-8ddb-555c4a364ade-plugins-config-volume\") pod \"payload-processing-5f78687d5b-6k58k\" (UID: \"d2a355c3-0703-422f-8ddb-555c4a364ade\") " pod="openshift-ingress/payload-processing-5f78687d5b-6k58k"
Oct 30 00:49:24.292752 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.292743    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a13e1eb5-b23d-49ac-9e1d-6b431e9f07fd-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-w45z8\" (UID: \"a13e1eb5-b23d-49ac-9e1d-6b431e9f07fd\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-w45z8"
Oct 30 00:49:24.293505 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.293484    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a13e1eb5-b23d-49ac-9e1d-6b431e9f07fd-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-w45z8\" (UID: \"a13e1eb5-b23d-49ac-9e1d-6b431e9f07fd\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-w45z8"
Oct 30 00:49:24.314184 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.314153    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7l52\" (UniqueName: \"kubernetes.io/projected/a13e1eb5-b23d-49ac-9e1d-6b431e9f07fd-kube-api-access-p7l52\") pod \"payload-pre-processing-7b98bb678c-w45z8\" (UID: \"a13e1eb5-b23d-49ac-9e1d-6b431e9f07fd\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-w45z8"
Oct 30 00:49:24.393529 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.393440    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a355c3-0703-422f-8ddb-555c4a364ade-plugins-config-volume\") pod \"payload-processing-5f78687d5b-6k58k\" (UID: \"d2a355c3-0703-422f-8ddb-555c4a364ade\") " pod="openshift-ingress/payload-processing-5f78687d5b-6k58k"
Oct 30 00:49:24.393670 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.393559    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqgwg\" (UniqueName: \"kubernetes.io/projected/d2a355c3-0703-422f-8ddb-555c4a364ade-kube-api-access-zqgwg\") pod \"payload-processing-5f78687d5b-6k58k\" (UID: \"d2a355c3-0703-422f-8ddb-555c4a364ade\") " pod="openshift-ingress/payload-processing-5f78687d5b-6k58k"
Oct 30 00:49:24.394331 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.394303    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a355c3-0703-422f-8ddb-555c4a364ade-plugins-config-volume\") pod \"payload-processing-5f78687d5b-6k58k\" (UID: \"d2a355c3-0703-422f-8ddb-555c4a364ade\") " pod="openshift-ingress/payload-processing-5f78687d5b-6k58k"
Oct 30 00:49:24.414908 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.414885    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqgwg\" (UniqueName: \"kubernetes.io/projected/d2a355c3-0703-422f-8ddb-555c4a364ade-kube-api-access-zqgwg\") pod \"payload-processing-5f78687d5b-6k58k\" (UID: \"d2a355c3-0703-422f-8ddb-555c4a364ade\") " pod="openshift-ingress/payload-processing-5f78687d5b-6k58k"
Oct 30 00:49:24.456372 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.456346    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w45z8"
Oct 30 00:49:24.551326 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.551297    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-6k58k"
Oct 30 00:49:24.749280 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.749250    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-w45z8"]
Oct 30 00:49:24.753917 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:49:24.753890    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13e1eb5_b23d_49ac_9e1d_6b431e9f07fd.slice/crio-0776a90fcf8ed629bccb3de830091ef2055fddab4ad5cff5138e0c4b24bacad1 WatchSource:0}: Error finding container 0776a90fcf8ed629bccb3de830091ef2055fddab4ad5cff5138e0c4b24bacad1: Status 404 returned error can't find the container with id 0776a90fcf8ed629bccb3de830091ef2055fddab4ad5cff5138e0c4b24bacad1
Oct 30 00:49:24.794673 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.794647    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w45z8" event={"ID":"a13e1eb5-b23d-49ac-9e1d-6b431e9f07fd","Type":"ContainerStarted","Data":"0776a90fcf8ed629bccb3de830091ef2055fddab4ad5cff5138e0c4b24bacad1"}
Oct 30 00:49:24.845582 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:24.845553    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-6k58k"]
Oct 30 00:49:24.850190 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:49:24.850165    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a355c3_0703_422f_8ddb_555c4a364ade.slice/crio-70ca879db27689c9d2215a4a4e84849e20e740bb56aa26aebc52f9d84500ae0a WatchSource:0}: Error finding container 70ca879db27689c9d2215a4a4e84849e20e740bb56aa26aebc52f9d84500ae0a: Status 404 returned error can't find the container with id 70ca879db27689c9d2215a4a4e84849e20e740bb56aa26aebc52f9d84500ae0a
Oct 30 00:49:25.800878 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:25.800839    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-6k58k" event={"ID":"d2a355c3-0703-422f-8ddb-555c4a364ade","Type":"ContainerStarted","Data":"70ca879db27689c9d2215a4a4e84849e20e740bb56aa26aebc52f9d84500ae0a"}
Oct 30 00:49:27.811005 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:27.810970    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w45z8" event={"ID":"a13e1eb5-b23d-49ac-9e1d-6b431e9f07fd","Type":"ContainerStarted","Data":"ad1813d9dd7463a4eb5be448df7d41805e1beef1c7417e51c5c632b04b6a9da3"}
Oct 30 00:49:27.811437 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:27.811039    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w45z8"
Oct 30 00:49:27.812434 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:27.812408    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-6k58k" event={"ID":"d2a355c3-0703-422f-8ddb-555c4a364ade","Type":"ContainerStarted","Data":"1f9a05f4ad95f315af6f188b166314a0a6f5a5381f4b41bdf4f258e83894dacc"}
Oct 30 00:49:27.812562 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:27.812545    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-6k58k"
Oct 30 00:49:27.848434 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:27.848391    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w45z8" podStartSLOduration=1.575043388 podStartE2EDuration="3.848379443s" totalImagesPullingTime="2.273336055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:27.847343499 +0000 UTC m=+480.185445411" watchObservedRunningTime="2026-10-30 00:49:27.848379443 +0000 UTC m=+480.186481353"
Oct 30 00:49:27.884738 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:27.884697    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-6k58k" podStartSLOduration=1.747540917 podStartE2EDuration="3.884683681s" totalImagesPullingTime="2.137142764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:27.883257451 +0000 UTC m=+480.221359362" watchObservedRunningTime="2026-10-30 00:49:27.884683681 +0000 UTC m=+480.222785989"
Oct 30 00:49:33.792288 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:33.792258    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-44vqk"
Oct 30 00:49:34.059218 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:34.059129    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-599679688f-rb5db"]
Oct 30 00:49:34.064096 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:34.064051    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-599679688f-rb5db"
Oct 30 00:49:34.084149 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:34.084122    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-599679688f-rb5db"]
Oct 30 00:49:34.176136 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:34.176100    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfebe80-0050-43c0-bff5-eecbfeb7ffb5-metrics-certs\") pod \"ai-gateway-controller-599679688f-rb5db\" (UID: \"1cfebe80-0050-43c0-bff5-eecbfeb7ffb5\") " pod="opendatahub/ai-gateway-controller-599679688f-rb5db"
Oct 30 00:49:34.176303 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:34.176149    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqbhw\" (UniqueName: \"kubernetes.io/projected/1cfebe80-0050-43c0-bff5-eecbfeb7ffb5-kube-api-access-kqbhw\") pod \"ai-gateway-controller-599679688f-rb5db\" (UID: \"1cfebe80-0050-43c0-bff5-eecbfeb7ffb5\") " pod="opendatahub/ai-gateway-controller-599679688f-rb5db"
Oct 30 00:49:34.276857 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:34.276827    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfebe80-0050-43c0-bff5-eecbfeb7ffb5-metrics-certs\") pod \"ai-gateway-controller-599679688f-rb5db\" (UID: \"1cfebe80-0050-43c0-bff5-eecbfeb7ffb5\") " pod="opendatahub/ai-gateway-controller-599679688f-rb5db"
Oct 30 00:49:34.276994 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:34.276861    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqbhw\" (UniqueName: \"kubernetes.io/projected/1cfebe80-0050-43c0-bff5-eecbfeb7ffb5-kube-api-access-kqbhw\") pod \"ai-gateway-controller-599679688f-rb5db\" (UID: \"1cfebe80-0050-43c0-bff5-eecbfeb7ffb5\") " pod="opendatahub/ai-gateway-controller-599679688f-rb5db"
Oct 30 00:49:34.279304 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:34.279277    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfebe80-0050-43c0-bff5-eecbfeb7ffb5-metrics-certs\") pod \"ai-gateway-controller-599679688f-rb5db\" (UID: \"1cfebe80-0050-43c0-bff5-eecbfeb7ffb5\") " pod="opendatahub/ai-gateway-controller-599679688f-rb5db"
Oct 30 00:49:34.299860 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:34.299831    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqbhw\" (UniqueName: \"kubernetes.io/projected/1cfebe80-0050-43c0-bff5-eecbfeb7ffb5-kube-api-access-kqbhw\") pod \"ai-gateway-controller-599679688f-rb5db\" (UID: \"1cfebe80-0050-43c0-bff5-eecbfeb7ffb5\") " pod="opendatahub/ai-gateway-controller-599679688f-rb5db"
Oct 30 00:49:34.376656 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:34.376575    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-599679688f-rb5db"
Oct 30 00:49:34.668920 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:34.668848    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-599679688f-rb5db"]
Oct 30 00:49:34.673838 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:49:34.673812    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cfebe80_0050_43c0_bff5_eecbfeb7ffb5.slice/crio-39363b0080e6dcd8884de849351c4b944ccf18d03d1ebe96cf03d02a4e201740 WatchSource:0}: Error finding container 39363b0080e6dcd8884de849351c4b944ccf18d03d1ebe96cf03d02a4e201740: Status 404 returned error can't find the container with id 39363b0080e6dcd8884de849351c4b944ccf18d03d1ebe96cf03d02a4e201740
Oct 30 00:49:34.843260 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:34.843227    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-599679688f-rb5db" event={"ID":"1cfebe80-0050-43c0-bff5-eecbfeb7ffb5","Type":"ContainerStarted","Data":"39363b0080e6dcd8884de849351c4b944ccf18d03d1ebe96cf03d02a4e201740"}
Oct 30 00:49:35.854267 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:35.854227    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-599679688f-rb5db" event={"ID":"1cfebe80-0050-43c0-bff5-eecbfeb7ffb5","Type":"ContainerStarted","Data":"42d8bfddebff88e41431a0cb5482fb6952b4ab66762293a4d2b2cb6fe079153f"}
Oct 30 00:49:35.854725 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:35.854315    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-599679688f-rb5db"
Oct 30 00:49:35.892125 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:35.892056    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-599679688f-rb5db" podStartSLOduration=1.383511209 podStartE2EDuration="1.892041753s" totalImagesPullingTime="508.530544ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:35.890591604 +0000 UTC m=+488.228693520" watchObservedRunningTime="2026-10-30 00:49:35.892041753 +0000 UTC m=+488.230143662"
Oct 30 00:49:38.819372 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:38.819342    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-6k58k"
Oct 30 00:49:38.819843 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:38.819404    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w45z8"
Oct 30 00:49:46.860304 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:46.860223    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-599679688f-rb5db"
Oct 30 00:49:46.926532 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:46.926501    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-44vqk"]
Oct 30 00:49:46.926755 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:46.926716    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-44vqk" podUID="b798b208-68eb-40ee-89b3-1a5f24654439" containerName="manager" containerID="cri-o://54b53e76654c643688c901c51b0ad7ae44bf11ad883f02146c4f2870ff42b3b4" gracePeriod=30
Oct 30 00:49:47.167840 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:47.167815    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-44vqk"
Oct 30 00:49:47.286380 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:47.286332    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b798b208-68eb-40ee-89b3-1a5f24654439-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b798b208-68eb-40ee-89b3-1a5f24654439-metrics-certs\") pod \"b798b208-68eb-40ee-89b3-1a5f24654439\" (UID: \"b798b208-68eb-40ee-89b3-1a5f24654439\") "
Oct 30 00:49:47.286538 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:47.286406    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b798b208-68eb-40ee-89b3-1a5f24654439-kube-api-access-njvvt\" (UniqueName: \"kubernetes.io/projected/b798b208-68eb-40ee-89b3-1a5f24654439-kube-api-access-njvvt\") pod \"b798b208-68eb-40ee-89b3-1a5f24654439\" (UID: \"b798b208-68eb-40ee-89b3-1a5f24654439\") "
Oct 30 00:49:47.288610 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:47.288579    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b798b208-68eb-40ee-89b3-1a5f24654439-kube-api-access-njvvt" pod "b798b208-68eb-40ee-89b3-1a5f24654439" (UID: "b798b208-68eb-40ee-89b3-1a5f24654439"). InnerVolumeSpecName "kube-api-access-njvvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:49:47.288723 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:47.288611    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b798b208-68eb-40ee-89b3-1a5f24654439-metrics-certs" pod "b798b208-68eb-40ee-89b3-1a5f24654439" (UID: "b798b208-68eb-40ee-89b3-1a5f24654439"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:49:47.387899 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:47.387859    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njvvt\" (UniqueName: \"kubernetes.io/projected/b798b208-68eb-40ee-89b3-1a5f24654439-kube-api-access-njvvt\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:49:47.387899 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:47.387896    4864 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b798b208-68eb-40ee-89b3-1a5f24654439-metrics-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:49:47.906866 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:47.906830    4864 generic.go:356] "Generic (PLEG): container finished" podID="b798b208-68eb-40ee-89b3-1a5f24654439" containerID="54b53e76654c643688c901c51b0ad7ae44bf11ad883f02146c4f2870ff42b3b4" exitCode=0
Oct 30 00:49:47.907283 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:47.906881    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-44vqk"
Oct 30 00:49:47.907283 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:47.906926    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-44vqk" event={"ID":"b798b208-68eb-40ee-89b3-1a5f24654439","Type":"ContainerDied","Data":"54b53e76654c643688c901c51b0ad7ae44bf11ad883f02146c4f2870ff42b3b4"}
Oct 30 00:49:47.907283 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:47.906979    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-44vqk" event={"ID":"b798b208-68eb-40ee-89b3-1a5f24654439","Type":"ContainerDied","Data":"d2559fa859c15b9611d29f5f38dba01641d78e7b0465ed6a8f04d3517f3cb8ec"}
Oct 30 00:49:47.907283 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:47.907000    4864 scope.go:122] "RemoveContainer" containerID="54b53e76654c643688c901c51b0ad7ae44bf11ad883f02146c4f2870ff42b3b4"
Oct 30 00:49:47.915306 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:47.915285    4864 scope.go:122] "RemoveContainer" containerID="54b53e76654c643688c901c51b0ad7ae44bf11ad883f02146c4f2870ff42b3b4"
Oct 30 00:49:47.915575 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:49:47.915555    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b53e76654c643688c901c51b0ad7ae44bf11ad883f02146c4f2870ff42b3b4\": container with ID starting with 54b53e76654c643688c901c51b0ad7ae44bf11ad883f02146c4f2870ff42b3b4 not found: ID does not exist" containerID="54b53e76654c643688c901c51b0ad7ae44bf11ad883f02146c4f2870ff42b3b4"
Oct 30 00:49:47.915642 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:47.915585    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b53e76654c643688c901c51b0ad7ae44bf11ad883f02146c4f2870ff42b3b4"} err="failed to get container status \"54b53e76654c643688c901c51b0ad7ae44bf11ad883f02146c4f2870ff42b3b4\": rpc error: code = NotFound desc = could not find container \"54b53e76654c643688c901c51b0ad7ae44bf11ad883f02146c4f2870ff42b3b4\": container with ID starting with 54b53e76654c643688c901c51b0ad7ae44bf11ad883f02146c4f2870ff42b3b4 not found: ID does not exist"
Oct 30 00:49:47.951194 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:47.951164    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-44vqk"]
Oct 30 00:49:47.961343 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:47.961317    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-44vqk"]
Oct 30 00:49:49.038028 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:49:49.037989    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:49:49.791027 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:49.790991    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b798b208-68eb-40ee-89b3-1a5f24654439" path="/var/lib/kubelet/pods/b798b208-68eb-40ee-89b3-1a5f24654439/volumes"
Oct 30 00:49:54.706758 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.706723    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"]
Oct 30 00:49:54.707369 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.707254    4864 cpu_manager.go:414] "containerMap: removing container" podUID="b798b208-68eb-40ee-89b3-1a5f24654439" containerName="manager"
Oct 30 00:49:54.707369 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.707272    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b798b208-68eb-40ee-89b3-1a5f24654439" containerName="manager"
Oct 30 00:49:54.707369 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.707353    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="b798b208-68eb-40ee-89b3-1a5f24654439" containerName="manager"
Oct 30 00:49:54.711866 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.711842    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"
Oct 30 00:49:54.735369 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.735169    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pmws7\""
Oct 30 00:49:54.735369 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.735231    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 00:49:54.735369 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.735343    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 00:49:54.735731 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.735710    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 00:49:54.736849 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.736829    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"]
Oct 30 00:49:54.850998 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.850961    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd5e277-4012-4026-896c-89c2aa9db931-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx\" (UID: \"bdd5e277-4012-4026-896c-89c2aa9db931\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"
Oct 30 00:49:54.851202 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.851006    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd5e277-4012-4026-896c-89c2aa9db931-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx\" (UID: \"bdd5e277-4012-4026-896c-89c2aa9db931\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"
Oct 30 00:49:54.851202 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.851034    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd5e277-4012-4026-896c-89c2aa9db931-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx\" (UID: \"bdd5e277-4012-4026-896c-89c2aa9db931\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"
Oct 30 00:49:54.851202 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.851115    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd5e277-4012-4026-896c-89c2aa9db931-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx\" (UID: \"bdd5e277-4012-4026-896c-89c2aa9db931\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"
Oct 30 00:49:54.851358 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.851207    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd5e277-4012-4026-896c-89c2aa9db931-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx\" (UID: \"bdd5e277-4012-4026-896c-89c2aa9db931\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"
Oct 30 00:49:54.851358 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.851305    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffxkm\" (UniqueName: \"kubernetes.io/projected/bdd5e277-4012-4026-896c-89c2aa9db931-kube-api-access-ffxkm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx\" (UID: \"bdd5e277-4012-4026-896c-89c2aa9db931\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"
Oct 30 00:49:54.952647 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.952615    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffxkm\" (UniqueName: \"kubernetes.io/projected/bdd5e277-4012-4026-896c-89c2aa9db931-kube-api-access-ffxkm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx\" (UID: \"bdd5e277-4012-4026-896c-89c2aa9db931\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"
Oct 30 00:49:54.952801 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.952663    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd5e277-4012-4026-896c-89c2aa9db931-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx\" (UID: \"bdd5e277-4012-4026-896c-89c2aa9db931\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"
Oct 30 00:49:54.952801 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.952783    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd5e277-4012-4026-896c-89c2aa9db931-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx\" (UID: \"bdd5e277-4012-4026-896c-89c2aa9db931\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"
Oct 30 00:49:54.952878 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.952821    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd5e277-4012-4026-896c-89c2aa9db931-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx\" (UID: \"bdd5e277-4012-4026-896c-89c2aa9db931\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"
Oct 30 00:49:54.952878 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.952854    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd5e277-4012-4026-896c-89c2aa9db931-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx\" (UID: \"bdd5e277-4012-4026-896c-89c2aa9db931\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"
Oct 30 00:49:54.952958 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.952885    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd5e277-4012-4026-896c-89c2aa9db931-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx\" (UID: \"bdd5e277-4012-4026-896c-89c2aa9db931\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"
Oct 30 00:49:54.953250 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.953225    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd5e277-4012-4026-896c-89c2aa9db931-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx\" (UID: \"bdd5e277-4012-4026-896c-89c2aa9db931\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"
Oct 30 00:49:54.953250 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.953246    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd5e277-4012-4026-896c-89c2aa9db931-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx\" (UID: \"bdd5e277-4012-4026-896c-89c2aa9db931\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"
Oct 30 00:49:54.953402 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.953291    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd5e277-4012-4026-896c-89c2aa9db931-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx\" (UID: \"bdd5e277-4012-4026-896c-89c2aa9db931\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"
Oct 30 00:49:54.954968 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.954950    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd5e277-4012-4026-896c-89c2aa9db931-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx\" (UID: \"bdd5e277-4012-4026-896c-89c2aa9db931\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"
Oct 30 00:49:54.955129 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.955113    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd5e277-4012-4026-896c-89c2aa9db931-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx\" (UID: \"bdd5e277-4012-4026-896c-89c2aa9db931\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"
Oct 30 00:49:54.974064 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:54.974007    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffxkm\" (UniqueName: \"kubernetes.io/projected/bdd5e277-4012-4026-896c-89c2aa9db931-kube-api-access-ffxkm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx\" (UID: \"bdd5e277-4012-4026-896c-89c2aa9db931\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"
Oct 30 00:49:55.024892 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:55.024865    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"
Oct 30 00:49:55.317748 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:55.317677    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"]
Oct 30 00:49:55.322359 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:49:55.322329    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd5e277_4012_4026_896c_89c2aa9db931.slice/crio-22b2846e1c5ca588f512bcd8b867018c19ae56f47dea9436cc0dfb0a880551e9 WatchSource:0}: Error finding container 22b2846e1c5ca588f512bcd8b867018c19ae56f47dea9436cc0dfb0a880551e9: Status 404 returned error can't find the container with id 22b2846e1c5ca588f512bcd8b867018c19ae56f47dea9436cc0dfb0a880551e9
Oct 30 00:49:55.943407 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:55.943372    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx" event={"ID":"bdd5e277-4012-4026-896c-89c2aa9db931","Type":"ContainerStarted","Data":"22b2846e1c5ca588f512bcd8b867018c19ae56f47dea9436cc0dfb0a880551e9"}
Oct 30 00:49:56.949199 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:56.949166    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx" event={"ID":"bdd5e277-4012-4026-896c-89c2aa9db931","Type":"ContainerStarted","Data":"f7f15cb019d7256867302913b490e53e3b3adeb51093a5411a1b4d9eaf5ba352"}
Oct 30 00:49:56.989864 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:56.989808    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx" podStartSLOduration=1.8950616139999998 podStartE2EDuration="2.989792915s" totalImagesPullingTime="1.094731301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:56.988097792 +0000 UTC m=+509.326199702" watchObservedRunningTime="2026-10-30 00:49:56.989792915 +0000 UTC m=+509.327894825"
Oct 30 00:49:57.025250 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:57.025214    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"
Oct 30 00:49:58.612916 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:58.612880    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"]
Oct 30 00:49:58.618466 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:58.618437    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"
Oct 30 00:49:58.633480 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:58.633446    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 00:49:58.645342 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:58.645313    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"]
Oct 30 00:49:58.784691 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:58.784655    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q58ws\" (UID: \"7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"
Oct 30 00:49:58.784837 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:58.784704    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q58ws\" (UID: \"7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"
Oct 30 00:49:58.784837 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:58.784731    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q58ws\" (UID: \"7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"
Oct 30 00:49:58.784931 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:58.784825    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q58ws\" (UID: \"7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"
Oct 30 00:49:58.784931 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:58.784858    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b22cj\" (UniqueName: \"kubernetes.io/projected/7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c-kube-api-access-b22cj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q58ws\" (UID: \"7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"
Oct 30 00:49:58.784995 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:58.784927    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q58ws\" (UID: \"7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"
Oct 30 00:49:58.885585 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:58.885488    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q58ws\" (UID: \"7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"
Oct 30 00:49:58.885585 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:58.885535    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b22cj\" (UniqueName: \"kubernetes.io/projected/7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c-kube-api-access-b22cj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q58ws\" (UID: \"7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"
Oct 30 00:49:58.885585 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:58.885568    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q58ws\" (UID: \"7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"
Oct 30 00:49:58.885870 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:58.885628    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q58ws\" (UID: \"7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"
Oct 30 00:49:58.885870 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:58.885685    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q58ws\" (UID: \"7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"
Oct 30 00:49:58.885870 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:58.885730    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q58ws\" (UID: \"7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"
Oct 30 00:49:58.886066 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:58.886037    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q58ws\" (UID: \"7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"
Oct 30 00:49:58.886264 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:58.886147    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q58ws\" (UID: \"7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"
Oct 30 00:49:58.886325 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:58.886257    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q58ws\" (UID: \"7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"
Oct 30 00:49:58.887873 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:58.887846    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q58ws\" (UID: \"7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"
Oct 30 00:49:58.888126 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:58.888110    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q58ws\" (UID: \"7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"
Oct 30 00:49:58.906749 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:58.906724    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b22cj\" (UniqueName: \"kubernetes.io/projected/7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c-kube-api-access-b22cj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q58ws\" (UID: \"7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"
Oct 30 00:49:58.931224 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:58.931185    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"
Oct 30 00:49:59.226996 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:59.226959    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"]
Oct 30 00:49:59.232390 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:49:59.232364    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce6cfb5_72b1_48d2_8311_10c69cf5dd2c.slice/crio-fa944ac47d110aadcc23ce0f1ecc9aa7c0eb429440a696c39bb24340f208fbfd WatchSource:0}: Error finding container fa944ac47d110aadcc23ce0f1ecc9aa7c0eb429440a696c39bb24340f208fbfd: Status 404 returned error can't find the container with id fa944ac47d110aadcc23ce0f1ecc9aa7c0eb429440a696c39bb24340f208fbfd
Oct 30 00:49:59.963895 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:59.963856    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws" event={"ID":"7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c","Type":"ContainerStarted","Data":"80cdbb4931ff63599b9cee975e38765f1fd81f04870af540cf5cc70aa36641c0"}
Oct 30 00:49:59.964281 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:49:59.963911    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws" event={"ID":"7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c","Type":"ContainerStarted","Data":"fa944ac47d110aadcc23ce0f1ecc9aa7c0eb429440a696c39bb24340f208fbfd"}
Oct 30 00:50:00.002637 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:00.002585    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws" podStartSLOduration=1.825587261 podStartE2EDuration="2.002570642s" totalImagesPullingTime="176.983381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:00.001857267 +0000 UTC m=+512.339959177" watchObservedRunningTime="2026-10-30 00:50:00.002570642 +0000 UTC m=+512.340672553"
Oct 30 00:50:00.931903 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:00.931852    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"
Oct 30 00:50:01.672467 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:01.672428    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"]
Oct 30 00:50:01.676271 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:01.676243    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"
Oct 30 00:50:01.689871 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:01.689840    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 00:50:01.702052 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:01.702026    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"]
Oct 30 00:50:01.811549 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:01.811513    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5btj\" (UniqueName: \"kubernetes.io/projected/ec811c9f-eaa8-4c9b-8b96-8b690fb63772-kube-api-access-r5btj\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6c89k\" (UID: \"ec811c9f-eaa8-4c9b-8b96-8b690fb63772\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"
Oct 30 00:50:01.811727 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:01.811553    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec811c9f-eaa8-4c9b-8b96-8b690fb63772-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6c89k\" (UID: \"ec811c9f-eaa8-4c9b-8b96-8b690fb63772\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"
Oct 30 00:50:01.811727 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:01.811638    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec811c9f-eaa8-4c9b-8b96-8b690fb63772-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6c89k\" (UID: \"ec811c9f-eaa8-4c9b-8b96-8b690fb63772\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"
Oct 30 00:50:01.811727 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:01.811684    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec811c9f-eaa8-4c9b-8b96-8b690fb63772-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6c89k\" (UID: \"ec811c9f-eaa8-4c9b-8b96-8b690fb63772\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"
Oct 30 00:50:01.811841 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:01.811768    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec811c9f-eaa8-4c9b-8b96-8b690fb63772-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6c89k\" (UID: \"ec811c9f-eaa8-4c9b-8b96-8b690fb63772\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"
Oct 30 00:50:01.811841 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:01.811792    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec811c9f-eaa8-4c9b-8b96-8b690fb63772-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6c89k\" (UID: \"ec811c9f-eaa8-4c9b-8b96-8b690fb63772\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"
Oct 30 00:50:01.913202 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:01.913165    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec811c9f-eaa8-4c9b-8b96-8b690fb63772-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6c89k\" (UID: \"ec811c9f-eaa8-4c9b-8b96-8b690fb63772\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"
Oct 30 00:50:01.913202 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:01.913204    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec811c9f-eaa8-4c9b-8b96-8b690fb63772-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6c89k\" (UID: \"ec811c9f-eaa8-4c9b-8b96-8b690fb63772\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"
Oct 30 00:50:01.913399 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:01.913247    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5btj\" (UniqueName: \"kubernetes.io/projected/ec811c9f-eaa8-4c9b-8b96-8b690fb63772-kube-api-access-r5btj\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6c89k\" (UID: \"ec811c9f-eaa8-4c9b-8b96-8b690fb63772\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"
Oct 30 00:50:01.913399 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:01.913262    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec811c9f-eaa8-4c9b-8b96-8b690fb63772-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6c89k\" (UID: \"ec811c9f-eaa8-4c9b-8b96-8b690fb63772\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"
Oct 30 00:50:01.913399 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:01.913282    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec811c9f-eaa8-4c9b-8b96-8b690fb63772-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6c89k\" (UID: \"ec811c9f-eaa8-4c9b-8b96-8b690fb63772\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"
Oct 30 00:50:01.913399 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:01.913297    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec811c9f-eaa8-4c9b-8b96-8b690fb63772-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6c89k\" (UID: \"ec811c9f-eaa8-4c9b-8b96-8b690fb63772\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"
Oct 30 00:50:01.913710 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:01.913679    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec811c9f-eaa8-4c9b-8b96-8b690fb63772-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6c89k\" (UID: \"ec811c9f-eaa8-4c9b-8b96-8b690fb63772\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"
Oct 30 00:50:01.913710 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:01.913704    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec811c9f-eaa8-4c9b-8b96-8b690fb63772-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6c89k\" (UID: \"ec811c9f-eaa8-4c9b-8b96-8b690fb63772\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"
Oct 30 00:50:01.913882 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:01.913727    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec811c9f-eaa8-4c9b-8b96-8b690fb63772-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6c89k\" (UID: \"ec811c9f-eaa8-4c9b-8b96-8b690fb63772\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"
Oct 30 00:50:01.915460 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:01.915433    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec811c9f-eaa8-4c9b-8b96-8b690fb63772-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6c89k\" (UID: \"ec811c9f-eaa8-4c9b-8b96-8b690fb63772\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"
Oct 30 00:50:01.915725 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:01.915707    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec811c9f-eaa8-4c9b-8b96-8b690fb63772-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6c89k\" (UID: \"ec811c9f-eaa8-4c9b-8b96-8b690fb63772\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"
Oct 30 00:50:01.934597 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:01.934523    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5btj\" (UniqueName: \"kubernetes.io/projected/ec811c9f-eaa8-4c9b-8b96-8b690fb63772-kube-api-access-r5btj\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6c89k\" (UID: \"ec811c9f-eaa8-4c9b-8b96-8b690fb63772\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"
Oct 30 00:50:01.990582 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:01.990544    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"
Oct 30 00:50:02.290091 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:02.290040    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"]
Oct 30 00:50:02.295207 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:50:02.295168    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec811c9f_eaa8_4c9b_8b96_8b690fb63772.slice/crio-1b47450d2b96588abc16f69c04aa8e114920588bd97c3bd6da76437d3b36ed08 WatchSource:0}: Error finding container 1b47450d2b96588abc16f69c04aa8e114920588bd97c3bd6da76437d3b36ed08: Status 404 returned error can't find the container with id 1b47450d2b96588abc16f69c04aa8e114920588bd97c3bd6da76437d3b36ed08
Oct 30 00:50:02.979278 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:02.979246    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k" event={"ID":"ec811c9f-eaa8-4c9b-8b96-8b690fb63772","Type":"ContainerStarted","Data":"262b67adf632e233e35ac05abe64cce94ce1e1d269e45e99c6223f315119a714"}
Oct 30 00:50:02.979645 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:02.979296    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k" event={"ID":"ec811c9f-eaa8-4c9b-8b96-8b690fb63772","Type":"ContainerStarted","Data":"1b47450d2b96588abc16f69c04aa8e114920588bd97c3bd6da76437d3b36ed08"}
Oct 30 00:50:02.991617 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:02.991594    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"
Oct 30 00:50:03.059228 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:03.059182    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k" podStartSLOduration=1.8620847390000002 podStartE2EDuration="2.059168281s" totalImagesPullingTime="197.083542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:03.057655594 +0000 UTC m=+515.395757515" watchObservedRunningTime="2026-10-30 00:50:03.059168281 +0000 UTC m=+515.397270190"
Oct 30 00:50:05.025858 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:05.025816    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"
Oct 30 00:50:05.038461 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:05.038435    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"
Oct 30 00:50:06.003617 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:06.003588    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx"
Oct 30 00:50:08.473345 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.473313    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"]
Oct 30 00:50:08.477120 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.477098    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"
Oct 30 00:50:08.500383 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.500341    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 00:50:08.504846 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.504822    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"]
Oct 30 00:50:08.575142 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.575109    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3c75ce0-9007-4481-a55a-e04106a30699-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss\" (UID: \"a3c75ce0-9007-4481-a55a-e04106a30699\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"
Oct 30 00:50:08.575292 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.575148    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk9z8\" (UniqueName: \"kubernetes.io/projected/a3c75ce0-9007-4481-a55a-e04106a30699-kube-api-access-mk9z8\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss\" (UID: \"a3c75ce0-9007-4481-a55a-e04106a30699\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"
Oct 30 00:50:08.575292 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.575269    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3c75ce0-9007-4481-a55a-e04106a30699-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss\" (UID: \"a3c75ce0-9007-4481-a55a-e04106a30699\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"
Oct 30 00:50:08.575394 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.575302    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3c75ce0-9007-4481-a55a-e04106a30699-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss\" (UID: \"a3c75ce0-9007-4481-a55a-e04106a30699\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"
Oct 30 00:50:08.575454 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.575411    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3c75ce0-9007-4481-a55a-e04106a30699-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss\" (UID: \"a3c75ce0-9007-4481-a55a-e04106a30699\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"
Oct 30 00:50:08.575507 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.575469    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3c75ce0-9007-4481-a55a-e04106a30699-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss\" (UID: \"a3c75ce0-9007-4481-a55a-e04106a30699\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"
Oct 30 00:50:08.676587 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.676549    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3c75ce0-9007-4481-a55a-e04106a30699-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss\" (UID: \"a3c75ce0-9007-4481-a55a-e04106a30699\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"
Oct 30 00:50:08.676587 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.676588    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3c75ce0-9007-4481-a55a-e04106a30699-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss\" (UID: \"a3c75ce0-9007-4481-a55a-e04106a30699\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"
Oct 30 00:50:08.676843 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.676616    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3c75ce0-9007-4481-a55a-e04106a30699-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss\" (UID: \"a3c75ce0-9007-4481-a55a-e04106a30699\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"
Oct 30 00:50:08.676843 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.676734    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3c75ce0-9007-4481-a55a-e04106a30699-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss\" (UID: \"a3c75ce0-9007-4481-a55a-e04106a30699\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"
Oct 30 00:50:08.676843 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.676798    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3c75ce0-9007-4481-a55a-e04106a30699-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss\" (UID: \"a3c75ce0-9007-4481-a55a-e04106a30699\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"
Oct 30 00:50:08.676843 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.676829    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk9z8\" (UniqueName: \"kubernetes.io/projected/a3c75ce0-9007-4481-a55a-e04106a30699-kube-api-access-mk9z8\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss\" (UID: \"a3c75ce0-9007-4481-a55a-e04106a30699\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"
Oct 30 00:50:08.677051 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.676997    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3c75ce0-9007-4481-a55a-e04106a30699-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss\" (UID: \"a3c75ce0-9007-4481-a55a-e04106a30699\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"
Oct 30 00:50:08.677130 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.677092    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3c75ce0-9007-4481-a55a-e04106a30699-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss\" (UID: \"a3c75ce0-9007-4481-a55a-e04106a30699\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"
Oct 30 00:50:08.677195 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.677151    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3c75ce0-9007-4481-a55a-e04106a30699-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss\" (UID: \"a3c75ce0-9007-4481-a55a-e04106a30699\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"
Oct 30 00:50:08.678900 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.678878    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3c75ce0-9007-4481-a55a-e04106a30699-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss\" (UID: \"a3c75ce0-9007-4481-a55a-e04106a30699\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"
Oct 30 00:50:08.679101 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.679085    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3c75ce0-9007-4481-a55a-e04106a30699-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss\" (UID: \"a3c75ce0-9007-4481-a55a-e04106a30699\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"
Oct 30 00:50:08.698443 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.698417    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk9z8\" (UniqueName: \"kubernetes.io/projected/a3c75ce0-9007-4481-a55a-e04106a30699-kube-api-access-mk9z8\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss\" (UID: \"a3c75ce0-9007-4481-a55a-e04106a30699\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"
Oct 30 00:50:08.793650 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.793565    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"
Oct 30 00:50:08.931592 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.931561    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"
Oct 30 00:50:08.943665 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:08.943635    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"
Oct 30 00:50:09.015941 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:09.015917    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q58ws"
Oct 30 00:50:09.128652 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:09.128561    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"]
Oct 30 00:50:09.132949 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:50:09.132926    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c75ce0_9007_4481_a55a_e04106a30699.slice/crio-9614eb82c40640b95bd3882339e3faf64fd66cdda06e57ff4e38e249f7589ec6 WatchSource:0}: Error finding container 9614eb82c40640b95bd3882339e3faf64fd66cdda06e57ff4e38e249f7589ec6: Status 404 returned error can't find the container with id 9614eb82c40640b95bd3882339e3faf64fd66cdda06e57ff4e38e249f7589ec6
Oct 30 00:50:10.011198 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:10.011156    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss" event={"ID":"a3c75ce0-9007-4481-a55a-e04106a30699","Type":"ContainerStarted","Data":"ff34fe0e3b61eb016c3d36f80e4b6638dbefc0eb4376ab4ff2fa4cce8e44ea1a"}
Oct 30 00:50:10.011578 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:10.011222    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss" event={"ID":"a3c75ce0-9007-4481-a55a-e04106a30699","Type":"ContainerStarted","Data":"9614eb82c40640b95bd3882339e3faf64fd66cdda06e57ff4e38e249f7589ec6"}
Oct 30 00:50:10.052289 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:10.052241    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss" podStartSLOduration=1.870754114 podStartE2EDuration="2.052227843s" totalImagesPullingTime="181.473729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:10.051094819 +0000 UTC m=+522.389196730" watchObservedRunningTime="2026-10-30 00:50:10.052227843 +0000 UTC m=+522.390329752"
Oct 30 00:50:10.793695 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:10.793658    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"
Oct 30 00:50:10.961458 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:10.961426    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"]
Oct 30 00:50:10.964990 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:10.964968    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:50:10.990634 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:10.990608    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 00:50:10.994396 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:10.994374    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba3a068-e4af-4bd1-b007-2f73f455f65c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh\" (UID: \"6ba3a068-e4af-4bd1-b007-2f73f455f65c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:50:10.994525 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:10.994400    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba3a068-e4af-4bd1-b007-2f73f455f65c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh\" (UID: \"6ba3a068-e4af-4bd1-b007-2f73f455f65c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:50:10.994525 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:10.994428    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba3a068-e4af-4bd1-b007-2f73f455f65c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh\" (UID: \"6ba3a068-e4af-4bd1-b007-2f73f455f65c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:50:10.994525 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:10.994449    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s98jx\" (UniqueName: \"kubernetes.io/projected/6ba3a068-e4af-4bd1-b007-2f73f455f65c-kube-api-access-s98jx\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh\" (UID: \"6ba3a068-e4af-4bd1-b007-2f73f455f65c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:50:10.994677 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:10.994624    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba3a068-e4af-4bd1-b007-2f73f455f65c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh\" (UID: \"6ba3a068-e4af-4bd1-b007-2f73f455f65c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:50:10.994729 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:10.994672    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba3a068-e4af-4bd1-b007-2f73f455f65c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh\" (UID: \"6ba3a068-e4af-4bd1-b007-2f73f455f65c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:50:11.013810 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:11.013785    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"]
Oct 30 00:50:11.095424 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:11.095325    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba3a068-e4af-4bd1-b007-2f73f455f65c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh\" (UID: \"6ba3a068-e4af-4bd1-b007-2f73f455f65c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:50:11.095424 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:11.095359    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba3a068-e4af-4bd1-b007-2f73f455f65c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh\" (UID: \"6ba3a068-e4af-4bd1-b007-2f73f455f65c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:50:11.095424 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:11.095404    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba3a068-e4af-4bd1-b007-2f73f455f65c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh\" (UID: \"6ba3a068-e4af-4bd1-b007-2f73f455f65c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:50:11.095710 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:11.095588    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba3a068-e4af-4bd1-b007-2f73f455f65c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh\" (UID: \"6ba3a068-e4af-4bd1-b007-2f73f455f65c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:50:11.095710 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:11.095665    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba3a068-e4af-4bd1-b007-2f73f455f65c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh\" (UID: \"6ba3a068-e4af-4bd1-b007-2f73f455f65c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:50:11.095710 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:11.095701    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s98jx\" (UniqueName: \"kubernetes.io/projected/6ba3a068-e4af-4bd1-b007-2f73f455f65c-kube-api-access-s98jx\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh\" (UID: \"6ba3a068-e4af-4bd1-b007-2f73f455f65c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:50:11.095855 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:11.095762    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba3a068-e4af-4bd1-b007-2f73f455f65c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh\" (UID: \"6ba3a068-e4af-4bd1-b007-2f73f455f65c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:50:11.095855 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:11.095803    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba3a068-e4af-4bd1-b007-2f73f455f65c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh\" (UID: \"6ba3a068-e4af-4bd1-b007-2f73f455f65c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:50:11.096039 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:11.096015    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba3a068-e4af-4bd1-b007-2f73f455f65c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh\" (UID: \"6ba3a068-e4af-4bd1-b007-2f73f455f65c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:50:11.097701 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:11.097676    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba3a068-e4af-4bd1-b007-2f73f455f65c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh\" (UID: \"6ba3a068-e4af-4bd1-b007-2f73f455f65c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:50:11.097977 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:11.097960    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba3a068-e4af-4bd1-b007-2f73f455f65c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh\" (UID: \"6ba3a068-e4af-4bd1-b007-2f73f455f65c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:50:11.124459 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:11.124430    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s98jx\" (UniqueName: \"kubernetes.io/projected/6ba3a068-e4af-4bd1-b007-2f73f455f65c-kube-api-access-s98jx\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh\" (UID: \"6ba3a068-e4af-4bd1-b007-2f73f455f65c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:50:11.292274 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:11.292239    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:50:11.592799 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:11.592743    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"]
Oct 30 00:50:11.596162 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:50:11.596132    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ba3a068_e4af_4bd1_b007_2f73f455f65c.slice/crio-627b9f7623e6a46fae87708e1aae7e2118b9ef399423547dc6ff8fe129bd4d1a WatchSource:0}: Error finding container 627b9f7623e6a46fae87708e1aae7e2118b9ef399423547dc6ff8fe129bd4d1a: Status 404 returned error can't find the container with id 627b9f7623e6a46fae87708e1aae7e2118b9ef399423547dc6ff8fe129bd4d1a
Oct 30 00:50:11.991425 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:11.991397    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"
Oct 30 00:50:12.004527 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:12.004503    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"
Oct 30 00:50:12.023174 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:12.023139    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh" event={"ID":"6ba3a068-e4af-4bd1-b007-2f73f455f65c","Type":"ContainerStarted","Data":"6754e44c21b1398681e5f9d95603b9a3d75440a0c4397d8b5cb240be83e27439"}
Oct 30 00:50:12.023585 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:12.023182    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh" event={"ID":"6ba3a068-e4af-4bd1-b007-2f73f455f65c","Type":"ContainerStarted","Data":"627b9f7623e6a46fae87708e1aae7e2118b9ef399423547dc6ff8fe129bd4d1a"}
Oct 30 00:50:12.034338 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:12.034316    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6c89k"
Oct 30 00:50:12.119632 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:12.119571    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh" podStartSLOduration=1.9448755119999999 podStartE2EDuration="2.119551649s" totalImagesPullingTime="174.676137ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:12.118554616 +0000 UTC m=+524.456656527" watchObservedRunningTime="2026-10-30 00:50:12.119551649 +0000 UTC m=+524.457653561"
Oct 30 00:50:12.292645 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:12.292568    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:50:18.794626 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:18.794591    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"
Oct 30 00:50:18.807429 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:18.807394    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"
Oct 30 00:50:19.041578 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:50:19.041540    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:50:19.068727 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:19.068659    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss"
Oct 30 00:50:21.293058 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:21.293023    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:50:21.305921 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:21.305891    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:50:22.083934 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:22.083907    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:50:27.460896 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.460858    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"]
Oct 30 00:50:27.464768 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.464747    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.485302 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.485275    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-spn7j\""
Oct 30 00:50:27.495819 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.495790    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 00:50:27.497624 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.497595    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"]
Oct 30 00:50:27.539509 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.539469    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.539509 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.539508    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.539730 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.539527    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.539730 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.539676    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxs45\" (UniqueName: \"kubernetes.io/projected/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-kube-api-access-qxs45\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.539730 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.539701    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.539730 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.539718    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.539885 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.539756    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.539885 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.539791    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.539885 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.539815    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.640875 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.640828    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.640875 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.640870    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.641173 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.640900    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.641173 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.640966    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.641173 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.641003    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.641173 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.641025    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.641173 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.641132    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxs45\" (UniqueName: \"kubernetes.io/projected/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-kube-api-access-qxs45\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.641173 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.641166    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.641453 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.641187    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.641521 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.641502    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.641631 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.641607    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.641631 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.641623    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.641718 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.641701    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.642050 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.642025    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.643457 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.643432    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.643609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.643589    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.662742 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.662703    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.662742 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.662737    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxs45\" (UniqueName: \"kubernetes.io/projected/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-kube-api-access-qxs45\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b6c79\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:27.790367 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.790251    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-spn7j\""
Oct 30 00:50:27.798748 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:27.798718    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:28.115484 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:28.115393    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"]
Oct 30 00:50:28.119678 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:50:28.119651    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fac8a72_32dd_4801_94bf_65e06ef2f8ff.slice/crio-b74fecf7bce1e041bfd15a74f0f8e73fe43b68dda72304c53b737c81b81a373e WatchSource:0}: Error finding container b74fecf7bce1e041bfd15a74f0f8e73fe43b68dda72304c53b737c81b81a373e: Status 404 returned error can't find the container with id b74fecf7bce1e041bfd15a74f0f8e73fe43b68dda72304c53b737c81b81a373e
Oct 30 00:50:29.106403 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:29.106358    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79" event={"ID":"8fac8a72-32dd-4801-94bf-65e06ef2f8ff","Type":"ContainerStarted","Data":"b74fecf7bce1e041bfd15a74f0f8e73fe43b68dda72304c53b737c81b81a373e"}
Oct 30 00:50:31.897729 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:31.897700    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:50:31.898007 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:31.897767    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:50:31.898007 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:31.897798    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:50:32.123621 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:32.123578    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79" event={"ID":"8fac8a72-32dd-4801-94bf-65e06ef2f8ff","Type":"ContainerStarted","Data":"1fbd64ce194902e1692dd89adc274e2b01a42ccefaf858f40b406501d71a72aa"}
Oct 30 00:50:32.185432 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:32.185336    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79" podStartSLOduration=1.410003216 podStartE2EDuration="5.185324142s" totalImagesPullingTime="3.775320926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:32.175051012 +0000 UTC m=+544.513152922" watchObservedRunningTime="2026-10-30 00:50:32.185324142 +0000 UTC m=+544.523426052"
Oct 30 00:50:32.799889 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:32.799849    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:32.801461 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:32.801434    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-b6c79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 30 00:50:32.801592 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:32.801482    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79" podUID="8fac8a72-32dd-4801-94bf-65e06ef2f8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 30 00:50:33.799668 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:33.799631    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-b6c79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 30 00:50:33.800166 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:33.799687    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79" podUID="8fac8a72-32dd-4801-94bf-65e06ef2f8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 30 00:50:34.799787 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:34.799748    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-b6c79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 30 00:50:34.800209 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:34.799817    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79" podUID="8fac8a72-32dd-4801-94bf-65e06ef2f8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 30 00:50:35.799980 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:35.799946    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-b6c79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 30 00:50:35.800393 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:35.799996    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79" podUID="8fac8a72-32dd-4801-94bf-65e06ef2f8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 30 00:50:36.799585 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:36.799548    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-b6c79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 30 00:50:36.799808 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:36.799602    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79" podUID="8fac8a72-32dd-4801-94bf-65e06ef2f8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 30 00:50:37.799265 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:37.799231    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-b6c79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 30 00:50:37.799643 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:37.799292    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79" podUID="8fac8a72-32dd-4801-94bf-65e06ef2f8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 30 00:50:38.799045 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:38.799013    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-b6c79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 30 00:50:38.799267 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:38.799097    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79" podUID="8fac8a72-32dd-4801-94bf-65e06ef2f8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 30 00:50:40.104114 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:40.104083    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:40.104989 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:40.104947    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:40.105646 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:50:40.105628    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:50:49.039001 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:50:49.038967    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:51:19.038813 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:51:19.038733    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:51:27.836747 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:51:27.836715    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 00:51:27.837755 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:51:27.837733    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 00:51:49.041878 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:51:49.041837    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:52:19.039360 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:52:19.039320    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:52:44.062645 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.062567    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"]
Oct 30 00:52:44.065550 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.065531    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.101311 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.101269    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"]
Oct 30 00:52:44.122275 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.122232    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq66q\" (UniqueName: \"kubernetes.io/projected/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-kube-api-access-nq66q\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.122451 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.122296    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-credential-socket\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.122451 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.122314    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istiod-ca-cert\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.122451 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.122394    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-workload-socket\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.122451 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.122436    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-podinfo\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.122595 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.122559    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-workload-certs\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.122595 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.122584    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-token\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.122668 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.122605    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-envoy\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.122668 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.122632    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-data\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.173594 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.173555    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-aaf98f1a-openshift-default-dockercfg-mcbm8\""
Oct 30 00:52:44.223874 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.223836    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-workload-certs\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.223874 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.223873    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-token\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.224122 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.223892    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-envoy\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.224122 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.223907    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-data\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.224122 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.223948    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq66q\" (UniqueName: \"kubernetes.io/projected/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-kube-api-access-nq66q\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.224122 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.223985    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-credential-socket\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.224122 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.223999    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istiod-ca-cert\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.224122 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.224024    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-workload-socket\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.224122 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.224051    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-podinfo\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.224506 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.224311    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-workload-certs\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.224590 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.224567    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-data\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.224653 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.224619    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-credential-socket\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.224715 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.224648    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-workload-socket\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.224837 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.224814    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istiod-ca-cert\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.226280 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.226260    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-envoy\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.226480 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.226463    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-podinfo\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.245490 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.245462    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-token\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.245597 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.245498    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq66q\" (UniqueName: \"kubernetes.io/projected/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-kube-api-access-nq66q\") pod \"e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") " pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.378976 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.378890    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:44.689750 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.689671    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"]
Oct 30 00:52:44.693900 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:52:44.693865    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c3fcfa_204a_43fc_b0fc_74658fd41ceb.slice/crio-9b4411f28d2d782e3788f9930129a00daaca138cc5c00b6716eaec1f2b42bb8f WatchSource:0}: Error finding container 9b4411f28d2d782e3788f9930129a00daaca138cc5c00b6716eaec1f2b42bb8f: Status 404 returned error can't find the container with id 9b4411f28d2d782e3788f9930129a00daaca138cc5c00b6716eaec1f2b42bb8f
Oct 30 00:52:44.696043 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.696009    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:52:44.696140 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.696100    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:52:44.696140 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.696132    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:52:44.729512 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:44.729339    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj" event={"ID":"89c3fcfa-204a-43fc-b0fc-74658fd41ceb","Type":"ContainerStarted","Data":"9b4411f28d2d782e3788f9930129a00daaca138cc5c00b6716eaec1f2b42bb8f"}
Oct 30 00:52:45.736035 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:45.735993    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj" event={"ID":"89c3fcfa-204a-43fc-b0fc-74658fd41ceb","Type":"ContainerStarted","Data":"31ad93f99084d1685591a97b39f2759348dabe2a3d66b0e09d425e8cf3da77d6"}
Oct 30 00:52:45.788849 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:45.788793    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj" podStartSLOduration=1.788776052 podStartE2EDuration="1.788776052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:45.787019894 +0000 UTC m=+678.125121808" watchObservedRunningTime="2026-10-30 00:52:45.788776052 +0000 UTC m=+678.126877964"
Oct 30 00:52:46.380108 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:46.380048    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:46.385121 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:46.385097    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:46.741665 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:46.741634    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:46.742764 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:46.742741    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:52:48.942766 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:48.942733    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"]
Oct 30 00:52:48.945991 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:48.945969    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:48.960493 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:48.960468    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5e61cc8d-openshift-default-dockercfg-q6lg8\""
Oct 30 00:52:48.977032 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:48.977001    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"]
Oct 30 00:52:49.039050 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:52:49.039017    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:52:49.064013 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.063973    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-token\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.064225 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.064094    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-workload-certs\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.064225 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.064137    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-workload-socket\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.064225 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.064204    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-envoy\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.064392 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.064250    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-podinfo\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.064392 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.064287    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istiod-ca-cert\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.064392 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.064365    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-credential-socket\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.064392 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.064384    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtlpd\" (UniqueName: \"kubernetes.io/projected/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-kube-api-access-jtlpd\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.064587 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.064444    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-data\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.112393 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.112362    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw"]
Oct 30 00:52:49.118366 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.118341    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw"
Oct 30 00:52:49.141954 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.141921    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw"]
Oct 30 00:52:49.160222 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.160189    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-aaf98f1a-serving-cert\""
Oct 30 00:52:49.165754 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.165719    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-envoy\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.165938 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.165774    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-podinfo\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.165938 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.165897    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istiod-ca-cert\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.166764 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.166733    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istiod-ca-cert\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.167178 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.167149    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-credential-socket\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.167308 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.166849    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-credential-socket\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.167308 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.167248    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtlpd\" (UniqueName: \"kubernetes.io/projected/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-kube-api-access-jtlpd\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.167785 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.167763    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-data\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.167892 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.167808    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-token\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.167892 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.167867    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-workload-certs\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.167997 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.167918    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-workload-socket\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.168501 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.168474    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-data\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.168822 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.168691    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-workload-certs\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.168913 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.168894    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-workload-socket\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.168973 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.168904    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-podinfo\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.169104 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.169066    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-envoy\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.195486 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.195401    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-token\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.195646 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.195518    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtlpd\" (UniqueName: \"kubernetes.io/projected/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-kube-api-access-jtlpd\") pod \"e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") " pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.263034 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.263002    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:49.269066 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.269032    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53924caf-da60-4fc7-8401-65d763f0a480-maas-api-tls\") pod \"maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw\" (UID: \"53924caf-da60-4fc7-8401-65d763f0a480\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw"
Oct 30 00:52:49.269208 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.269095    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53924caf-da60-4fc7-8401-65d763f0a480-openshift-service-ca\") pod \"maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw\" (UID: \"53924caf-da60-4fc7-8401-65d763f0a480\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw"
Oct 30 00:52:49.269250 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.269195    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9cqs\" (UniqueName: \"kubernetes.io/projected/53924caf-da60-4fc7-8401-65d763f0a480-kube-api-access-x9cqs\") pod \"maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw\" (UID: \"53924caf-da60-4fc7-8401-65d763f0a480\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw"
Oct 30 00:52:49.269289 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.269239    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53924caf-da60-4fc7-8401-65d763f0a480-metrics-certs\") pod \"maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw\" (UID: \"53924caf-da60-4fc7-8401-65d763f0a480\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw"
Oct 30 00:52:49.370509 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.370464    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53924caf-da60-4fc7-8401-65d763f0a480-metrics-certs\") pod \"maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw\" (UID: \"53924caf-da60-4fc7-8401-65d763f0a480\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw"
Oct 30 00:52:49.370674 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.370613    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53924caf-da60-4fc7-8401-65d763f0a480-maas-api-tls\") pod \"maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw\" (UID: \"53924caf-da60-4fc7-8401-65d763f0a480\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw"
Oct 30 00:52:49.370674 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.370632    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53924caf-da60-4fc7-8401-65d763f0a480-openshift-service-ca\") pod \"maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw\" (UID: \"53924caf-da60-4fc7-8401-65d763f0a480\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw"
Oct 30 00:52:49.370768 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.370676    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9cqs\" (UniqueName: \"kubernetes.io/projected/53924caf-da60-4fc7-8401-65d763f0a480-kube-api-access-x9cqs\") pod \"maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw\" (UID: \"53924caf-da60-4fc7-8401-65d763f0a480\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw"
Oct 30 00:52:49.371351 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.371317    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53924caf-da60-4fc7-8401-65d763f0a480-openshift-service-ca\") pod \"maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw\" (UID: \"53924caf-da60-4fc7-8401-65d763f0a480\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw"
Oct 30 00:52:49.372965 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.372943    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53924caf-da60-4fc7-8401-65d763f0a480-metrics-certs\") pod \"maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw\" (UID: \"53924caf-da60-4fc7-8401-65d763f0a480\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw"
Oct 30 00:52:49.373174 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.373150    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53924caf-da60-4fc7-8401-65d763f0a480-maas-api-tls\") pod \"maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw\" (UID: \"53924caf-da60-4fc7-8401-65d763f0a480\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw"
Oct 30 00:52:49.392746 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.392715    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9cqs\" (UniqueName: \"kubernetes.io/projected/53924caf-da60-4fc7-8401-65d763f0a480-kube-api-access-x9cqs\") pod \"maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw\" (UID: \"53924caf-da60-4fc7-8401-65d763f0a480\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw"
Oct 30 00:52:49.435561 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.435532    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw"
Oct 30 00:52:49.579091 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.579044    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"]
Oct 30 00:52:49.582801 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:52:49.582776    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0225b6be_e7fc_4b20_a524_9ce6e6135c1f.slice/crio-5014f8ad9c496de8a08c398fdfaafc4e0534e97f2238b194d7dabe613507d8c2 WatchSource:0}: Error finding container 5014f8ad9c496de8a08c398fdfaafc4e0534e97f2238b194d7dabe613507d8c2: Status 404 returned error can't find the container with id 5014f8ad9c496de8a08c398fdfaafc4e0534e97f2238b194d7dabe613507d8c2
Oct 30 00:52:49.584987 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.584960    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:52:49.585056 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.585021    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:52:49.585056 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.585049    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:52:49.741909 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.741870    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw"]
Oct 30 00:52:49.747995 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:52:49.747962    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53924caf_da60_4fc7_8401_65d763f0a480.slice/crio-a527acdfff80427b70b63a43e954496c0afa513c320b98c8d60c1f67ea08ac3d WatchSource:0}: Error finding container a527acdfff80427b70b63a43e954496c0afa513c320b98c8d60c1f67ea08ac3d: Status 404 returned error can't find the container with id a527acdfff80427b70b63a43e954496c0afa513c320b98c8d60c1f67ea08ac3d
Oct 30 00:52:49.749435 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.749414    4864 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:52:49.757000 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.756974    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw" event={"ID":"53924caf-da60-4fc7-8401-65d763f0a480","Type":"ContainerStarted","Data":"a527acdfff80427b70b63a43e954496c0afa513c320b98c8d60c1f67ea08ac3d"}
Oct 30 00:52:49.758431 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.758407    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w" event={"ID":"0225b6be-e7fc-4b20-a524-9ce6e6135c1f","Type":"ContainerStarted","Data":"74e0ae4b73aca38da9accc308916f718d7cd9cbc579d2a6a1ca8ed14db283510"}
Oct 30 00:52:49.758539 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.758447    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w" event={"ID":"0225b6be-e7fc-4b20-a524-9ce6e6135c1f","Type":"ContainerStarted","Data":"5014f8ad9c496de8a08c398fdfaafc4e0534e97f2238b194d7dabe613507d8c2"}
Oct 30 00:52:49.800561 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:49.800514    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w" podStartSLOduration=1.8005002700000001 podStartE2EDuration="1.80050027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:49.799104666 +0000 UTC m=+682.137206568" watchObservedRunningTime="2026-10-30 00:52:49.80050027 +0000 UTC m=+682.138602180"
Oct 30 00:52:50.263424 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:50.263389    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:50.268235 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:50.268209    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:50.764735 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:50.764700    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw" event={"ID":"53924caf-da60-4fc7-8401-65d763f0a480","Type":"ContainerStarted","Data":"8e2db9c4b97ab1b55b6b407c08792d793e02284e630d2a9e66292ead4c4719f0"}
Oct 30 00:52:50.765017 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:50.764999    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw"
Oct 30 00:52:50.765106 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:50.765045    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:50.765750 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:50.765732    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:52:50.813318 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:50.813246    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw" podStartSLOduration=1.421549584 podStartE2EDuration="1.813227207s" totalImagesPullingTime="391.677623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:50.810942296 +0000 UTC m=+683.149044207" watchObservedRunningTime="2026-10-30 00:52:50.813227207 +0000 UTC m=+683.151329118"
Oct 30 00:52:52.469208 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:52.469176    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22"]
Oct 30 00:52:52.471799 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:52.471781    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22"
Oct 30 00:52:52.492941 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:52.492911    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22"]
Oct 30 00:52:52.506646 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:52.506611    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-bbc42836\""
Oct 30 00:52:52.553874 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:52.553841    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-bbc42836-dockercfg-sc44k\""
Oct 30 00:52:52.604530 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:52.604493    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26d9d81-1722-4d9f-97c8-f4065502affc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22\" (UID: \"a26d9d81-1722-4d9f-97c8-f4065502affc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22"
Oct 30 00:52:52.604698 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:52.604554    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhhw9\" (UniqueName: \"kubernetes.io/projected/a26d9d81-1722-4d9f-97c8-f4065502affc-kube-api-access-vhhw9\") pod \"payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22\" (UID: \"a26d9d81-1722-4d9f-97c8-f4065502affc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22"
Oct 30 00:52:52.706057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:52.706013    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26d9d81-1722-4d9f-97c8-f4065502affc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22\" (UID: \"a26d9d81-1722-4d9f-97c8-f4065502affc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22"
Oct 30 00:52:52.706288 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:52.706132    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhw9\" (UniqueName: \"kubernetes.io/projected/a26d9d81-1722-4d9f-97c8-f4065502affc-kube-api-access-vhhw9\") pod \"payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22\" (UID: \"a26d9d81-1722-4d9f-97c8-f4065502affc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22"
Oct 30 00:52:52.706913 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:52.706888    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26d9d81-1722-4d9f-97c8-f4065502affc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22\" (UID: \"a26d9d81-1722-4d9f-97c8-f4065502affc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22"
Oct 30 00:52:52.730499 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:52.730432    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhhw9\" (UniqueName: \"kubernetes.io/projected/a26d9d81-1722-4d9f-97c8-f4065502affc-kube-api-access-vhhw9\") pod \"payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22\" (UID: \"a26d9d81-1722-4d9f-97c8-f4065502affc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22"
Oct 30 00:52:52.783609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:52.783580    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22"
Oct 30 00:52:53.076953 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:53.076863    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22"]
Oct 30 00:52:53.081881 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:52:53.081847    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda26d9d81_1722_4d9f_97c8_f4065502affc.slice/crio-48d0cffbf36e74bf813667ead812d384047e04add3ff0306b2220e08eb55c357 WatchSource:0}: Error finding container 48d0cffbf36e74bf813667ead812d384047e04add3ff0306b2220e08eb55c357: Status 404 returned error can't find the container with id 48d0cffbf36e74bf813667ead812d384047e04add3ff0306b2220e08eb55c357
Oct 30 00:52:53.780784 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:53.780748    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22" event={"ID":"a26d9d81-1722-4d9f-97c8-f4065502affc","Type":"ContainerStarted","Data":"a997853c317b50499c5e9fe19235e2f2ab1234f8d79965057dd7e171300a1c8b"}
Oct 30 00:52:53.781234 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:53.780852    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22"
Oct 30 00:52:53.781234 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:53.780881    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22" event={"ID":"a26d9d81-1722-4d9f-97c8-f4065502affc","Type":"ContainerStarted","Data":"48d0cffbf36e74bf813667ead812d384047e04add3ff0306b2220e08eb55c357"}
Oct 30 00:52:53.821088 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:53.819930    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22" podStartSLOduration=1.819908482 podStartE2EDuration="1.819908482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:53.816858236 +0000 UTC m=+686.154960144" watchObservedRunningTime="2026-10-30 00:52:53.819908482 +0000 UTC m=+686.158010394"
Oct 30 00:52:54.883782 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:54.883752    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s"]
Oct 30 00:52:54.886278 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:54.886262    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s"
Oct 30 00:52:54.899123 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:54.899084    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-bbc42836-serving-cert\""
Oct 30 00:52:54.912802 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:54.912764    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s"]
Oct 30 00:52:55.027775 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:55.027745    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp5wk\" (UniqueName: \"kubernetes.io/projected/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-kube-api-access-tp5wk\") pod \"maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s\" (UID: \"fe95fb9a-7b34-4851-bc85-b30d170bf5b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s"
Oct 30 00:52:55.027957 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:55.027795    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-maas-api-tls\") pod \"maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s\" (UID: \"fe95fb9a-7b34-4851-bc85-b30d170bf5b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s"
Oct 30 00:52:55.027957 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:55.027823    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-metrics-certs\") pod \"maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s\" (UID: \"fe95fb9a-7b34-4851-bc85-b30d170bf5b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s"
Oct 30 00:52:55.028068 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:55.027945    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-openshift-service-ca\") pod \"maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s\" (UID: \"fe95fb9a-7b34-4851-bc85-b30d170bf5b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s"
Oct 30 00:52:55.129489 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:55.129456    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp5wk\" (UniqueName: \"kubernetes.io/projected/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-kube-api-access-tp5wk\") pod \"maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s\" (UID: \"fe95fb9a-7b34-4851-bc85-b30d170bf5b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s"
Oct 30 00:52:55.129662 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:55.129510    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-maas-api-tls\") pod \"maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s\" (UID: \"fe95fb9a-7b34-4851-bc85-b30d170bf5b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s"
Oct 30 00:52:55.129662 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:55.129529    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-metrics-certs\") pod \"maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s\" (UID: \"fe95fb9a-7b34-4851-bc85-b30d170bf5b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s"
Oct 30 00:52:55.129662 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:55.129569    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-openshift-service-ca\") pod \"maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s\" (UID: \"fe95fb9a-7b34-4851-bc85-b30d170bf5b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s"
Oct 30 00:52:55.130355 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:55.130332    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-openshift-service-ca\") pod \"maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s\" (UID: \"fe95fb9a-7b34-4851-bc85-b30d170bf5b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s"
Oct 30 00:52:55.131990 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:55.131965    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-metrics-certs\") pod \"maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s\" (UID: \"fe95fb9a-7b34-4851-bc85-b30d170bf5b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s"
Oct 30 00:52:55.132085 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:55.132042    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-maas-api-tls\") pod \"maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s\" (UID: \"fe95fb9a-7b34-4851-bc85-b30d170bf5b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s"
Oct 30 00:52:55.150867 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:55.150809    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp5wk\" (UniqueName: \"kubernetes.io/projected/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-kube-api-access-tp5wk\") pod \"maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s\" (UID: \"fe95fb9a-7b34-4851-bc85-b30d170bf5b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s"
Oct 30 00:52:55.197623 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:55.197587    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s"
Oct 30 00:52:55.510597 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:55.510547    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s"]
Oct 30 00:52:55.514630 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:52:55.514604    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe95fb9a_7b34_4851_bc85_b30d170bf5b5.slice/crio-86a2185190bc63d027de27c11c4d7ddefbaa4005c51fad8946de6974833c5e28 WatchSource:0}: Error finding container 86a2185190bc63d027de27c11c4d7ddefbaa4005c51fad8946de6974833c5e28: Status 404 returned error can't find the container with id 86a2185190bc63d027de27c11c4d7ddefbaa4005c51fad8946de6974833c5e28
Oct 30 00:52:55.792407 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:55.792337    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s" event={"ID":"fe95fb9a-7b34-4851-bc85-b30d170bf5b5","Type":"ContainerStarted","Data":"86a2185190bc63d027de27c11c4d7ddefbaa4005c51fad8946de6974833c5e28"}
Oct 30 00:52:56.785958 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:56.785920    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw"
Oct 30 00:52:56.798393 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:56.798360    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s" event={"ID":"fe95fb9a-7b34-4851-bc85-b30d170bf5b5","Type":"ContainerStarted","Data":"62cff40db10e7f79d253327a04d414b128865fa8ca7db0df9670f45889be1e10"}
Oct 30 00:52:56.798527 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:56.798447    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s"
Oct 30 00:52:56.885305 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:52:56.885241    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s" podStartSLOduration=2.59067685 podStartE2EDuration="2.885218521s" totalImagesPullingTime="294.541671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:56.883690806 +0000 UTC m=+689.221792730" watchObservedRunningTime="2026-10-30 00:52:56.885218521 +0000 UTC m=+689.223320432"
Oct 30 00:53:02.572444 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.572402    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs"]
Oct 30 00:53:02.576049 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.576023    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs"
Oct 30 00:53:02.596504 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.596472    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs"]
Oct 30 00:53:02.604294 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.604270    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b06528d1\""
Oct 30 00:53:02.604504 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.604482    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-b06528d1-dockercfg-4npkc\""
Oct 30 00:53:02.647150 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.647119    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7"]
Oct 30 00:53:02.649687 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.649664    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7"
Oct 30 00:53:02.671461 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.671436    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7"]
Oct 30 00:53:02.728014 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.727978    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fvwl\" (UniqueName: \"kubernetes.io/projected/ac105c72-22ed-456a-807b-90f4d672da14-kube-api-access-2fvwl\") pod \"payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs\" (UID: \"ac105c72-22ed-456a-807b-90f4d672da14\") " pod="openshift-ingress/payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs"
Oct 30 00:53:02.728211 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.728045    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac105c72-22ed-456a-807b-90f4d672da14-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs\" (UID: \"ac105c72-22ed-456a-807b-90f4d672da14\") " pod="openshift-ingress/payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs"
Oct 30 00:53:02.783230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.783192    4864 patch_prober.go:29] interesting pod/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:53:02.783376 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.783271    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj" podUID="89c3fcfa-204a-43fc-b0fc-74658fd41ceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:02.809248 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.809201    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s"
Oct 30 00:53:02.829573 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.829508    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fvwl\" (UniqueName: \"kubernetes.io/projected/ac105c72-22ed-456a-807b-90f4d672da14-kube-api-access-2fvwl\") pod \"payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs\" (UID: \"ac105c72-22ed-456a-807b-90f4d672da14\") " pod="openshift-ingress/payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs"
Oct 30 00:53:02.829573 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.829550    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac105c72-22ed-456a-807b-90f4d672da14-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs\" (UID: \"ac105c72-22ed-456a-807b-90f4d672da14\") " pod="openshift-ingress/payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs"
Oct 30 00:53:02.829729 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.829589    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/481bc47b-b2f6-4d18-a747-413ac025eebb-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7\" (UID: \"481bc47b-b2f6-4d18-a747-413ac025eebb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7"
Oct 30 00:53:02.829779 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.829756    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmzc6\" (UniqueName: \"kubernetes.io/projected/481bc47b-b2f6-4d18-a747-413ac025eebb-kube-api-access-wmzc6\") pod \"payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7\" (UID: \"481bc47b-b2f6-4d18-a747-413ac025eebb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7"
Oct 30 00:53:02.830323 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.830305    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac105c72-22ed-456a-807b-90f4d672da14-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs\" (UID: \"ac105c72-22ed-456a-807b-90f4d672da14\") " pod="openshift-ingress/payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs"
Oct 30 00:53:02.853955 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.853915    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fvwl\" (UniqueName: \"kubernetes.io/projected/ac105c72-22ed-456a-807b-90f4d672da14-kube-api-access-2fvwl\") pod \"payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs\" (UID: \"ac105c72-22ed-456a-807b-90f4d672da14\") " pod="openshift-ingress/payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs"
Oct 30 00:53:02.888543 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.888385    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs"
Oct 30 00:53:02.931103 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.931037    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmzc6\" (UniqueName: \"kubernetes.io/projected/481bc47b-b2f6-4d18-a747-413ac025eebb-kube-api-access-wmzc6\") pod \"payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7\" (UID: \"481bc47b-b2f6-4d18-a747-413ac025eebb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7"
Oct 30 00:53:02.931263 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.931180    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/481bc47b-b2f6-4d18-a747-413ac025eebb-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7\" (UID: \"481bc47b-b2f6-4d18-a747-413ac025eebb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7"
Oct 30 00:53:02.932000 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.931972    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/481bc47b-b2f6-4d18-a747-413ac025eebb-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7\" (UID: \"481bc47b-b2f6-4d18-a747-413ac025eebb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7"
Oct 30 00:53:02.955747 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.955708    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmzc6\" (UniqueName: \"kubernetes.io/projected/481bc47b-b2f6-4d18-a747-413ac025eebb-kube-api-access-wmzc6\") pod \"payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7\" (UID: \"481bc47b-b2f6-4d18-a747-413ac025eebb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7"
Oct 30 00:53:02.962201 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:02.961316    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7"
Oct 30 00:53:03.188771 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:03.188678    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs"]
Oct 30 00:53:03.192690 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:53:03.192665    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac105c72_22ed_456a_807b_90f4d672da14.slice/crio-19216143d7ffa30eb28c43c911768107f74ddbd438817a0db9e0900b25cf67db WatchSource:0}: Error finding container 19216143d7ffa30eb28c43c911768107f74ddbd438817a0db9e0900b25cf67db: Status 404 returned error can't find the container with id 19216143d7ffa30eb28c43c911768107f74ddbd438817a0db9e0900b25cf67db
Oct 30 00:53:03.276805 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:03.276779    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7"]
Oct 30 00:53:03.280403 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:53:03.280380    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod481bc47b_b2f6_4d18_a747_413ac025eebb.slice/crio-5661a204f40f90329d719a7d60d63789393403ba672359b674f2acc1032a06d3 WatchSource:0}: Error finding container 5661a204f40f90329d719a7d60d63789393403ba672359b674f2acc1032a06d3: Status 404 returned error can't find the container with id 5661a204f40f90329d719a7d60d63789393403ba672359b674f2acc1032a06d3
Oct 30 00:53:03.834087 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:03.834046    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7" event={"ID":"481bc47b-b2f6-4d18-a747-413ac025eebb","Type":"ContainerStarted","Data":"fb6f3da2c98f37a617a485290fbcbcb9a3410a099cc64296fa96ba55e25b5f0a"}
Oct 30 00:53:03.834538 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:03.834125    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7"
Oct 30 00:53:03.834538 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:03.834150    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7" event={"ID":"481bc47b-b2f6-4d18-a747-413ac025eebb","Type":"ContainerStarted","Data":"5661a204f40f90329d719a7d60d63789393403ba672359b674f2acc1032a06d3"}
Oct 30 00:53:03.835604 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:03.835577    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs" event={"ID":"ac105c72-22ed-456a-807b-90f4d672da14","Type":"ContainerStarted","Data":"82cfa5c1482962d531bb4cf3c035844385fd1fc5cb91f8ef65845b0c6b813fe5"}
Oct 30 00:53:03.835685 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:03.835622    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs" event={"ID":"ac105c72-22ed-456a-807b-90f4d672da14","Type":"ContainerStarted","Data":"19216143d7ffa30eb28c43c911768107f74ddbd438817a0db9e0900b25cf67db"}
Oct 30 00:53:03.835685 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:03.835674    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs"
Oct 30 00:53:03.883351 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:03.878670    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7" podStartSLOduration=1.878655645 podStartE2EDuration="1.878655645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:03.875323897 +0000 UTC m=+696.213425809" watchObservedRunningTime="2026-10-30 00:53:03.878655645 +0000 UTC m=+696.216757554"
Oct 30 00:53:03.913983 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:03.913933    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs" podStartSLOduration=1.913919792 podStartE2EDuration="1.913919792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:03.912456137 +0000 UTC m=+696.250558059" watchObservedRunningTime="2026-10-30 00:53:03.913919792 +0000 UTC m=+696.252021701"
Oct 30 00:53:04.239600 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:04.239568    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj"]
Oct 30 00:53:04.242448 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:04.242431    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj"
Oct 30 00:53:04.255514 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:04.255492    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3cc211d9\""
Oct 30 00:53:04.255611 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:04.255535    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3cc211d9-dockercfg-w99dw\""
Oct 30 00:53:04.263363 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:04.263341    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj"]
Oct 30 00:53:04.344665 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:04.344625    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4g2m\" (UniqueName: \"kubernetes.io/projected/8588a088-9405-4acf-be5b-73aefb0c2402-kube-api-access-s4g2m\") pod \"payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj\" (UID: \"8588a088-9405-4acf-be5b-73aefb0c2402\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj"
Oct 30 00:53:04.344822 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:04.344691    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8588a088-9405-4acf-be5b-73aefb0c2402-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj\" (UID: \"8588a088-9405-4acf-be5b-73aefb0c2402\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj"
Oct 30 00:53:04.445549 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:04.445514    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8588a088-9405-4acf-be5b-73aefb0c2402-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj\" (UID: \"8588a088-9405-4acf-be5b-73aefb0c2402\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj"
Oct 30 00:53:04.445714 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:04.445615    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4g2m\" (UniqueName: \"kubernetes.io/projected/8588a088-9405-4acf-be5b-73aefb0c2402-kube-api-access-s4g2m\") pod \"payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj\" (UID: \"8588a088-9405-4acf-be5b-73aefb0c2402\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj"
Oct 30 00:53:04.446217 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:04.446198    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8588a088-9405-4acf-be5b-73aefb0c2402-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj\" (UID: \"8588a088-9405-4acf-be5b-73aefb0c2402\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj"
Oct 30 00:53:04.467125 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:04.467093    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4g2m\" (UniqueName: \"kubernetes.io/projected/8588a088-9405-4acf-be5b-73aefb0c2402-kube-api-access-s4g2m\") pod \"payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj\" (UID: \"8588a088-9405-4acf-be5b-73aefb0c2402\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj"
Oct 30 00:53:04.553613 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:04.553530    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj"
Oct 30 00:53:04.788566 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:04.788539    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22"
Oct 30 00:53:04.844461 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:04.844387    4864 generic.go:356] "Generic (PLEG): container finished" podID="89c3fcfa-204a-43fc-b0fc-74658fd41ceb" containerID="31ad93f99084d1685591a97b39f2759348dabe2a3d66b0e09d425e8cf3da77d6" exitCode=0
Oct 30 00:53:04.844943 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:04.844454    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj" event={"ID":"89c3fcfa-204a-43fc-b0fc-74658fd41ceb","Type":"ContainerDied","Data":"31ad93f99084d1685591a97b39f2759348dabe2a3d66b0e09d425e8cf3da77d6"}
Oct 30 00:53:04.845027 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:04.845007    4864 scope.go:122] "RemoveContainer" containerID="31ad93f99084d1685591a97b39f2759348dabe2a3d66b0e09d425e8cf3da77d6"
Oct 30 00:53:04.846629 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:04.846601    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:53:04.846761 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:04.846702    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:53:04.846761 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:04.846738    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:53:04.850740 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:04.850717    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj"]
Oct 30 00:53:04.860624 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:53:04.860597    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8588a088_9405_4acf_be5b_73aefb0c2402.slice/crio-11a69a392203e67209d83827d0d27fe3ca2b80cb13e1c8177e386f7a4071fedb WatchSource:0}: Error finding container 11a69a392203e67209d83827d0d27fe3ca2b80cb13e1c8177e386f7a4071fedb: Status 404 returned error can't find the container with id 11a69a392203e67209d83827d0d27fe3ca2b80cb13e1c8177e386f7a4071fedb
Oct 30 00:53:05.850996 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:05.850961    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj" event={"ID":"8588a088-9405-4acf-be5b-73aefb0c2402","Type":"ContainerStarted","Data":"00d82f64aaf7526273cd1c5e70a81d22fde3f7ba29d55591f1eeb8f89464f11a"}
Oct 30 00:53:05.851437 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:05.851014    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj" event={"ID":"8588a088-9405-4acf-be5b-73aefb0c2402","Type":"ContainerStarted","Data":"11a69a392203e67209d83827d0d27fe3ca2b80cb13e1c8177e386f7a4071fedb"}
Oct 30 00:53:05.851437 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:05.851055    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj"
Oct 30 00:53:05.853125 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:05.853101    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj" event={"ID":"89c3fcfa-204a-43fc-b0fc-74658fd41ceb","Type":"ContainerStarted","Data":"4be419010029f4635d7334a2f93f0c6d2f4853843d0a114c26355fab89938c4c"}
Oct 30 00:53:05.902869 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:05.902814    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj" podStartSLOduration=1.902801004 podStartE2EDuration="1.902801004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:05.901146804 +0000 UTC m=+698.239248739" watchObservedRunningTime="2026-10-30 00:53:05.902801004 +0000 UTC m=+698.240902913"
Oct 30 00:53:06.380455 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:06.380422    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:53:06.380789 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:06.380765    4864 patch_prober.go:29] interesting pod/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 00:53:06.380951 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:06.380926    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj" podUID="89c3fcfa-204a-43fc-b0fc-74658fd41ceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 00:53:06.472097 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:06.470743    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q"]
Oct 30 00:53:06.483894 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:06.483864    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q"
Oct 30 00:53:06.488097 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:06.487771    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q"]
Oct 30 00:53:06.488248 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:06.488119    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfw57\" (UniqueName: \"kubernetes.io/projected/564bc142-7b17-4f9f-bc53-b01980123954-kube-api-access-gfw57\") pod \"maas-api-e2e-ait-3cc211d9-55b788c775-khc7q\" (UID: \"564bc142-7b17-4f9f-bc53-b01980123954\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q"
Oct 30 00:53:06.488248 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:06.488172    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/564bc142-7b17-4f9f-bc53-b01980123954-metrics-certs\") pod \"maas-api-e2e-ait-3cc211d9-55b788c775-khc7q\" (UID: \"564bc142-7b17-4f9f-bc53-b01980123954\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q"
Oct 30 00:53:06.488248 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:06.488211    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/564bc142-7b17-4f9f-bc53-b01980123954-maas-api-tls\") pod \"maas-api-e2e-ait-3cc211d9-55b788c775-khc7q\" (UID: \"564bc142-7b17-4f9f-bc53-b01980123954\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q"
Oct 30 00:53:06.488476 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:06.488438    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/564bc142-7b17-4f9f-bc53-b01980123954-openshift-service-ca\") pod \"maas-api-e2e-ait-3cc211d9-55b788c775-khc7q\" (UID: \"564bc142-7b17-4f9f-bc53-b01980123954\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q"
Oct 30 00:53:06.496755 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:06.496731    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3cc211d9-serving-cert\""
Oct 30 00:53:06.588970 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:06.588933    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfw57\" (UniqueName: \"kubernetes.io/projected/564bc142-7b17-4f9f-bc53-b01980123954-kube-api-access-gfw57\") pod \"maas-api-e2e-ait-3cc211d9-55b788c775-khc7q\" (UID: \"564bc142-7b17-4f9f-bc53-b01980123954\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q"
Oct 30 00:53:06.588970 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:06.588971    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/564bc142-7b17-4f9f-bc53-b01980123954-metrics-certs\") pod \"maas-api-e2e-ait-3cc211d9-55b788c775-khc7q\" (UID: \"564bc142-7b17-4f9f-bc53-b01980123954\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q"
Oct 30 00:53:06.589236 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:06.588993    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/564bc142-7b17-4f9f-bc53-b01980123954-maas-api-tls\") pod \"maas-api-e2e-ait-3cc211d9-55b788c775-khc7q\" (UID: \"564bc142-7b17-4f9f-bc53-b01980123954\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q"
Oct 30 00:53:06.589236 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:06.589051    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/564bc142-7b17-4f9f-bc53-b01980123954-openshift-service-ca\") pod \"maas-api-e2e-ait-3cc211d9-55b788c775-khc7q\" (UID: \"564bc142-7b17-4f9f-bc53-b01980123954\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q"
Oct 30 00:53:06.589718 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:06.589691    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/564bc142-7b17-4f9f-bc53-b01980123954-openshift-service-ca\") pod \"maas-api-e2e-ait-3cc211d9-55b788c775-khc7q\" (UID: \"564bc142-7b17-4f9f-bc53-b01980123954\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q"
Oct 30 00:53:06.591484 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:06.591450    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/564bc142-7b17-4f9f-bc53-b01980123954-metrics-certs\") pod \"maas-api-e2e-ait-3cc211d9-55b788c775-khc7q\" (UID: \"564bc142-7b17-4f9f-bc53-b01980123954\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q"
Oct 30 00:53:06.591576 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:06.591549    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/564bc142-7b17-4f9f-bc53-b01980123954-maas-api-tls\") pod \"maas-api-e2e-ait-3cc211d9-55b788c775-khc7q\" (UID: \"564bc142-7b17-4f9f-bc53-b01980123954\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q"
Oct 30 00:53:06.610729 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:06.610703    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfw57\" (UniqueName: \"kubernetes.io/projected/564bc142-7b17-4f9f-bc53-b01980123954-kube-api-access-gfw57\") pod \"maas-api-e2e-ait-3cc211d9-55b788c775-khc7q\" (UID: \"564bc142-7b17-4f9f-bc53-b01980123954\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q"
Oct 30 00:53:06.800710 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:06.800675    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q"
Oct 30 00:53:07.105275 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:07.105188    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q"]
Oct 30 00:53:07.109018 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:53:07.108992    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod564bc142_7b17_4f9f_bc53_b01980123954.slice/crio-45fc849a169373072a0085a10c6a8ae72da977e2199921c88211bd48d6eebea5 WatchSource:0}: Error finding container 45fc849a169373072a0085a10c6a8ae72da977e2199921c88211bd48d6eebea5: Status 404 returned error can't find the container with id 45fc849a169373072a0085a10c6a8ae72da977e2199921c88211bd48d6eebea5
Oct 30 00:53:07.379577 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:07.379477    4864 patch_prober.go:29] interesting pod/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 00:53:07.379850 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:07.379815    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj" podUID="89c3fcfa-204a-43fc-b0fc-74658fd41ceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 00:53:07.897285 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:07.897247    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q" event={"ID":"564bc142-7b17-4f9f-bc53-b01980123954","Type":"ContainerStarted","Data":"a2b23601cac3a3b718878a8f5bcf2a6a22c0baada8221c791b21b9f327b67efd"}
Oct 30 00:53:07.897472 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:07.897305    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q" event={"ID":"564bc142-7b17-4f9f-bc53-b01980123954","Type":"ContainerStarted","Data":"45fc849a169373072a0085a10c6a8ae72da977e2199921c88211bd48d6eebea5"}
Oct 30 00:53:07.897472 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:07.897353    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q"
Oct 30 00:53:07.937299 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:07.937246    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q" podStartSLOduration=1.635772601 podStartE2EDuration="1.937232864s" totalImagesPullingTime="301.460263ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:07.935164202 +0000 UTC m=+700.273266125" watchObservedRunningTime="2026-10-30 00:53:07.937232864 +0000 UTC m=+700.275334768"
Oct 30 00:53:08.281121 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:08.280817    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx"]
Oct 30 00:53:08.284852 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:08.284818    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx"
Oct 30 00:53:08.297598 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:08.297573    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx"]
Oct 30 00:53:08.298799 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:08.298781    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-b06528d1-serving-cert\""
Oct 30 00:53:08.380470 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:08.380433    4864 patch_prober.go:29] interesting pod/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 00:53:08.380738 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:08.380708    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj" podUID="89c3fcfa-204a-43fc-b0fc-74658fd41ceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 00:53:08.404874 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:08.404838    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9h7c\" (UniqueName: \"kubernetes.io/projected/40d05fec-21fd-4c3e-9cce-f63766dbd08c-kube-api-access-v9h7c\") pod \"maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx\" (UID: \"40d05fec-21fd-4c3e-9cce-f63766dbd08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx"
Oct 30 00:53:08.404874 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:08.404876    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40d05fec-21fd-4c3e-9cce-f63766dbd08c-maas-api-tls\") pod \"maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx\" (UID: \"40d05fec-21fd-4c3e-9cce-f63766dbd08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx"
Oct 30 00:53:08.405090 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:08.405031    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d05fec-21fd-4c3e-9cce-f63766dbd08c-openshift-service-ca\") pod \"maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx\" (UID: \"40d05fec-21fd-4c3e-9cce-f63766dbd08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx"
Oct 30 00:53:08.405205 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:08.405180    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40d05fec-21fd-4c3e-9cce-f63766dbd08c-metrics-certs\") pod \"maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx\" (UID: \"40d05fec-21fd-4c3e-9cce-f63766dbd08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx"
Oct 30 00:53:08.506244 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:08.506191    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40d05fec-21fd-4c3e-9cce-f63766dbd08c-metrics-certs\") pod \"maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx\" (UID: \"40d05fec-21fd-4c3e-9cce-f63766dbd08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx"
Oct 30 00:53:08.506244 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:08.506247    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9h7c\" (UniqueName: \"kubernetes.io/projected/40d05fec-21fd-4c3e-9cce-f63766dbd08c-kube-api-access-v9h7c\") pod \"maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx\" (UID: \"40d05fec-21fd-4c3e-9cce-f63766dbd08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx"
Oct 30 00:53:08.506477 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:08.506277    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40d05fec-21fd-4c3e-9cce-f63766dbd08c-maas-api-tls\") pod \"maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx\" (UID: \"40d05fec-21fd-4c3e-9cce-f63766dbd08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx"
Oct 30 00:53:08.506477 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:08.506397    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d05fec-21fd-4c3e-9cce-f63766dbd08c-openshift-service-ca\") pod \"maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx\" (UID: \"40d05fec-21fd-4c3e-9cce-f63766dbd08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx"
Oct 30 00:53:08.507061 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:08.507037    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d05fec-21fd-4c3e-9cce-f63766dbd08c-openshift-service-ca\") pod \"maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx\" (UID: \"40d05fec-21fd-4c3e-9cce-f63766dbd08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx"
Oct 30 00:53:08.508684 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:08.508657    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40d05fec-21fd-4c3e-9cce-f63766dbd08c-maas-api-tls\") pod \"maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx\" (UID: \"40d05fec-21fd-4c3e-9cce-f63766dbd08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx"
Oct 30 00:53:08.508771 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:08.508736    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40d05fec-21fd-4c3e-9cce-f63766dbd08c-metrics-certs\") pod \"maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx\" (UID: \"40d05fec-21fd-4c3e-9cce-f63766dbd08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx"
Oct 30 00:53:08.527964 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:08.527935    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9h7c\" (UniqueName: \"kubernetes.io/projected/40d05fec-21fd-4c3e-9cce-f63766dbd08c-kube-api-access-v9h7c\") pod \"maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx\" (UID: \"40d05fec-21fd-4c3e-9cce-f63766dbd08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx"
Oct 30 00:53:08.598321 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:08.598240    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx"
Oct 30 00:53:08.897230 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:08.897147    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx"]
Oct 30 00:53:08.901549 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:53:08.901521    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d05fec_21fd_4c3e_9cce_f63766dbd08c.slice/crio-3c6201270b6495312319a46cd181ac3567ba59bbc3cc3c2b8536798df114c826 WatchSource:0}: Error finding container 3c6201270b6495312319a46cd181ac3567ba59bbc3cc3c2b8536798df114c826: Status 404 returned error can't find the container with id 3c6201270b6495312319a46cd181ac3567ba59bbc3cc3c2b8536798df114c826
Oct 30 00:53:09.382110 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:09.380431    4864 patch_prober.go:29] interesting pod/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 00:53:09.382110 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:09.380493    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj" podUID="89c3fcfa-204a-43fc-b0fc-74658fd41ceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 00:53:09.909433 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:09.909394    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx" event={"ID":"40d05fec-21fd-4c3e-9cce-f63766dbd08c","Type":"ContainerStarted","Data":"aa18d2bea6ae6230d69d84bde6ba1804442116d85e1e1ef952cb13da6aba5b9e"}
Oct 30 00:53:09.909616 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:09.909457    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx" event={"ID":"40d05fec-21fd-4c3e-9cce-f63766dbd08c","Type":"ContainerStarted","Data":"3c6201270b6495312319a46cd181ac3567ba59bbc3cc3c2b8536798df114c826"}
Oct 30 00:53:09.909616 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:09.909502    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx"
Oct 30 00:53:09.947492 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:09.947447    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx" podStartSLOduration=1.628975023 podStartE2EDuration="1.947434494s" totalImagesPullingTime="318.459471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:09.947002565 +0000 UTC m=+702.285104476" watchObservedRunningTime="2026-10-30 00:53:09.947434494 +0000 UTC m=+702.285536415"
Oct 30 00:53:09.990841 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:09.990803    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"]
Oct 30 00:53:09.993528 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:09.993510    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"
Oct 30 00:53:10.009238 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:10.009214    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5e61cc8d-serving-cert\""
Oct 30 00:53:10.015206 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:10.015183    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"]
Oct 30 00:53:10.019377 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:10.019350    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wcjc\" (UniqueName: \"kubernetes.io/projected/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-kube-api-access-7wcjc\") pod \"maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b\" (UID: \"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"
Oct 30 00:53:10.019490 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:10.019420    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-maas-api-tls\") pod \"maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b\" (UID: \"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"
Oct 30 00:53:10.019563 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:10.019527    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-metrics-certs\") pod \"maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b\" (UID: \"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"
Oct 30 00:53:10.019563 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:10.019556    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-openshift-service-ca\") pod \"maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b\" (UID: \"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"
Oct 30 00:53:10.120065 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:10.120036    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-maas-api-tls\") pod \"maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b\" (UID: \"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"
Oct 30 00:53:10.120216 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:10.120103    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-metrics-certs\") pod \"maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b\" (UID: \"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"
Oct 30 00:53:10.120216 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:10.120125    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-openshift-service-ca\") pod \"maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b\" (UID: \"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"
Oct 30 00:53:10.120216 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:53:10.120161    4864 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-serving-cert: secret "maas-api-e2e-ait-5e61cc8d-serving-cert" not found
Oct 30 00:53:10.120216 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:10.120202    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wcjc\" (UniqueName: \"kubernetes.io/projected/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-kube-api-access-7wcjc\") pod \"maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b\" (UID: \"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"
Oct 30 00:53:10.120448 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:53:10.120217    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-maas-api-tls podName:96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93 nodeName:}" failed. No retries permitted until 2026-10-30 00:53:10.620202286 +0000 UTC m=+702.958304175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-maas-api-tls") pod "maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b" (UID: "96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93") : secret "maas-api-e2e-ait-5e61cc8d-serving-cert" not found
Oct 30 00:53:10.120873 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:10.120848    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-openshift-service-ca\") pod \"maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b\" (UID: \"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"
Oct 30 00:53:10.122542 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:10.122523    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-metrics-certs\") pod \"maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b\" (UID: \"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"
Oct 30 00:53:10.145227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:10.145197    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wcjc\" (UniqueName: \"kubernetes.io/projected/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-kube-api-access-7wcjc\") pod \"maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b\" (UID: \"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"
Oct 30 00:53:10.380532 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:10.380497    4864 patch_prober.go:29] interesting pod/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 00:53:10.380807 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:10.380773    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj" podUID="89c3fcfa-204a-43fc-b0fc-74658fd41ceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 00:53:10.625528 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:10.625482    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-maas-api-tls\") pod \"maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b\" (UID: \"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"
Oct 30 00:53:10.627974 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:10.627950    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-maas-api-tls\") pod \"maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b\" (UID: \"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"
Oct 30 00:53:10.908380 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:10.908335    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"
Oct 30 00:53:11.211503 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:11.211465    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"]
Oct 30 00:53:11.215648 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:53:11.215617    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96afb4dd_cfa3_4ce9_a74d_4c6ecfea0e93.slice/crio-4a69d6ded2e4bdd3afc5cde3b734f11c1fe412596377d66e7a5e1ca63128c496 WatchSource:0}: Error finding container 4a69d6ded2e4bdd3afc5cde3b734f11c1fe412596377d66e7a5e1ca63128c496: Status 404 returned error can't find the container with id 4a69d6ded2e4bdd3afc5cde3b734f11c1fe412596377d66e7a5e1ca63128c496
Oct 30 00:53:11.382939 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:11.382908    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:53:11.383321 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:11.383299    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:53:11.383995 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:11.383976    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:53:11.923154 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:11.923055    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b" event={"ID":"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93","Type":"ContainerStarted","Data":"63f40ed8ca8587bbc39b58189917103eab1c51cac39f1f1d5efcb96fbda45e33"}
Oct 30 00:53:11.923612 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:11.923299    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"
Oct 30 00:53:11.923612 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:11.923325    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b" event={"ID":"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93","Type":"ContainerStarted","Data":"4a69d6ded2e4bdd3afc5cde3b734f11c1fe412596377d66e7a5e1ca63128c496"}
Oct 30 00:53:11.961770 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:11.961724    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b" podStartSLOduration=2.628811696 podStartE2EDuration="2.961707671s" totalImagesPullingTime="332.895975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:11.961303598 +0000 UTC m=+704.299405508" watchObservedRunningTime="2026-10-30 00:53:11.961707671 +0000 UTC m=+704.299809588"
Oct 30 00:53:13.908297 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:13.908268    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q"
Oct 30 00:53:14.847938 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:14.847911    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs"
Oct 30 00:53:14.848126 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:14.848059    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7"
Oct 30 00:53:15.921407 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:15.921381    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx"
Oct 30 00:53:16.893887 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:16.893861    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj"
Oct 30 00:53:17.934698 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:17.934672    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"
Oct 30 00:53:19.038810 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:53:19.038777    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:53:27.037886 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:27.037854    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22"]
Oct 30 00:53:28.028584 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:28.028551    4864 generic.go:356] "Generic (PLEG): container finished" podID="0225b6be-e7fc-4b20-a524-9ce6e6135c1f" containerID="74e0ae4b73aca38da9accc308916f718d7cd9cbc579d2a6a1ca8ed14db283510" exitCode=0
Oct 30 00:53:28.028755 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:28.028626    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w" event={"ID":"0225b6be-e7fc-4b20-a524-9ce6e6135c1f","Type":"ContainerDied","Data":"74e0ae4b73aca38da9accc308916f718d7cd9cbc579d2a6a1ca8ed14db283510"}
Oct 30 00:53:28.029022 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:28.029006    4864 scope.go:122] "RemoveContainer" containerID="74e0ae4b73aca38da9accc308916f718d7cd9cbc579d2a6a1ca8ed14db283510"
Oct 30 00:53:28.030397 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:28.030366    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:53:28.030490 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:28.030433    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:53:28.030490 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:28.030469    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:53:29.035723 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:29.035688    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w" event={"ID":"0225b6be-e7fc-4b20-a524-9ce6e6135c1f","Type":"ContainerStarted","Data":"41cbecbc2d99a2ec292415718c442171ae8be9a16835b54fe9194d5307454a7e"}
Oct 30 00:53:29.268095 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:29.266269    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:53:29.268095 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:29.266721    4864 patch_prober.go:29] interesting pod/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 00:53:29.268095 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:29.266772    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w" podUID="0225b6be-e7fc-4b20-a524-9ce6e6135c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 00:53:30.275097 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:30.274467    4864 patch_prober.go:29] interesting pod/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 00:53:30.275097 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:30.274561    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w" podUID="0225b6be-e7fc-4b20-a524-9ce6e6135c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 00:53:31.274096 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:31.270504    4864 patch_prober.go:29] interesting pod/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 00:53:31.274096 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:31.270577    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w" podUID="0225b6be-e7fc-4b20-a524-9ce6e6135c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 00:53:32.039103 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:32.039043    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22" podUID="a26d9d81-1722-4d9f-97c8-f4065502affc" containerName="payload-pre-processing" containerID="cri-o://a997853c317b50499c5e9fe19235e2f2ab1234f8d79965057dd7e171300a1c8b" gracePeriod=25
Oct 30 00:53:32.281114 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:32.266528    4864 patch_prober.go:29] interesting pod/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 00:53:32.281114 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:32.266606    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w" podUID="0225b6be-e7fc-4b20-a524-9ce6e6135c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 00:53:32.353717 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:32.353691    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22"
Oct 30 00:53:32.451825 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:32.451788    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a26d9d81-1722-4d9f-97c8-f4065502affc-kube-api-access-vhhw9\" (UniqueName: \"kubernetes.io/projected/a26d9d81-1722-4d9f-97c8-f4065502affc-kube-api-access-vhhw9\") pod \"a26d9d81-1722-4d9f-97c8-f4065502affc\" (UID: \"a26d9d81-1722-4d9f-97c8-f4065502affc\") "
Oct 30 00:53:32.452024 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:32.451897    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a26d9d81-1722-4d9f-97c8-f4065502affc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26d9d81-1722-4d9f-97c8-f4065502affc-plugins-config-volume\") pod \"a26d9d81-1722-4d9f-97c8-f4065502affc\" (UID: \"a26d9d81-1722-4d9f-97c8-f4065502affc\") "
Oct 30 00:53:32.452370 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:32.452334    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a26d9d81-1722-4d9f-97c8-f4065502affc-plugins-config-volume" pod "a26d9d81-1722-4d9f-97c8-f4065502affc" (UID: "a26d9d81-1722-4d9f-97c8-f4065502affc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:53:32.454685 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:32.454656    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a26d9d81-1722-4d9f-97c8-f4065502affc-kube-api-access-vhhw9" pod "a26d9d81-1722-4d9f-97c8-f4065502affc" (UID: "a26d9d81-1722-4d9f-97c8-f4065502affc"). InnerVolumeSpecName "kube-api-access-vhhw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:53:32.553511 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:32.553474    4864 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26d9d81-1722-4d9f-97c8-f4065502affc-plugins-config-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:32.553511 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:32.553505    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhhw9\" (UniqueName: \"kubernetes.io/projected/a26d9d81-1722-4d9f-97c8-f4065502affc-kube-api-access-vhhw9\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:33.083202 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:33.083163    4864 generic.go:356] "Generic (PLEG): container finished" podID="a26d9d81-1722-4d9f-97c8-f4065502affc" containerID="a997853c317b50499c5e9fe19235e2f2ab1234f8d79965057dd7e171300a1c8b" exitCode=0
Oct 30 00:53:33.083695 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:33.083218    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22"
Oct 30 00:53:33.083695 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:33.083247    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22" event={"ID":"a26d9d81-1722-4d9f-97c8-f4065502affc","Type":"ContainerDied","Data":"a997853c317b50499c5e9fe19235e2f2ab1234f8d79965057dd7e171300a1c8b"}
Oct 30 00:53:33.083695 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:33.083311    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22" event={"ID":"a26d9d81-1722-4d9f-97c8-f4065502affc","Type":"ContainerDied","Data":"48d0cffbf36e74bf813667ead812d384047e04add3ff0306b2220e08eb55c357"}
Oct 30 00:53:33.083695 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:33.083341    4864 scope.go:122] "RemoveContainer" containerID="a997853c317b50499c5e9fe19235e2f2ab1234f8d79965057dd7e171300a1c8b"
Oct 30 00:53:33.094728 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:33.094707    4864 scope.go:122] "RemoveContainer" containerID="a997853c317b50499c5e9fe19235e2f2ab1234f8d79965057dd7e171300a1c8b"
Oct 30 00:53:33.095006 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:53:33.094989    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a997853c317b50499c5e9fe19235e2f2ab1234f8d79965057dd7e171300a1c8b\": container with ID starting with a997853c317b50499c5e9fe19235e2f2ab1234f8d79965057dd7e171300a1c8b not found: ID does not exist" containerID="a997853c317b50499c5e9fe19235e2f2ab1234f8d79965057dd7e171300a1c8b"
Oct 30 00:53:33.095052 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:33.095015    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a997853c317b50499c5e9fe19235e2f2ab1234f8d79965057dd7e171300a1c8b"} err="failed to get container status \"a997853c317b50499c5e9fe19235e2f2ab1234f8d79965057dd7e171300a1c8b\": rpc error: code = NotFound desc = could not find container \"a997853c317b50499c5e9fe19235e2f2ab1234f8d79965057dd7e171300a1c8b\": container with ID starting with a997853c317b50499c5e9fe19235e2f2ab1234f8d79965057dd7e171300a1c8b not found: ID does not exist"
Oct 30 00:53:33.134025 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:33.134000    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22"]
Oct 30 00:53:33.138747 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:33.138715    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bbc42836-6cddfcdd7f-pdb22"]
Oct 30 00:53:33.265503 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:33.265467    4864 patch_prober.go:29] interesting pod/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 00:53:33.265771 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:33.265740    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w" podUID="0225b6be-e7fc-4b20-a524-9ce6e6135c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 00:53:33.792423 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:33.792386    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a26d9d81-1722-4d9f-97c8-f4065502affc" path="/var/lib/kubelet/pods/a26d9d81-1722-4d9f-97c8-f4065502affc/volumes"
Oct 30 00:53:34.269254 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:34.269222    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:53:34.269743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:34.269580    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:53:34.270497 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:34.270476    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:53:34.946108 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:34.946060    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s"]
Oct 30 00:53:34.946339 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:34.946316    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s" podUID="fe95fb9a-7b34-4851-bc85-b30d170bf5b5" containerName="maas-api" containerID="cri-o://62cff40db10e7f79d253327a04d414b128865fa8ca7db0df9670f45889be1e10" gracePeriod=30
Oct 30 00:53:35.098556 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:35.098527    4864 generic.go:356] "Generic (PLEG): container finished" podID="fe95fb9a-7b34-4851-bc85-b30d170bf5b5" containerID="62cff40db10e7f79d253327a04d414b128865fa8ca7db0df9670f45889be1e10" exitCode=0
Oct 30 00:53:35.098715 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:35.098603    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s" event={"ID":"fe95fb9a-7b34-4851-bc85-b30d170bf5b5","Type":"ContainerDied","Data":"62cff40db10e7f79d253327a04d414b128865fa8ca7db0df9670f45889be1e10"}
Oct 30 00:53:35.186667 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:35.186641    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s"
Oct 30 00:53:35.299609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:35.299569    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-maas-api-tls\") pod \"fe95fb9a-7b34-4851-bc85-b30d170bf5b5\" (UID: \"fe95fb9a-7b34-4851-bc85-b30d170bf5b5\") "
Oct 30 00:53:35.300142 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:35.299620    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-kube-api-access-tp5wk\" (UniqueName: \"kubernetes.io/projected/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-kube-api-access-tp5wk\") pod \"fe95fb9a-7b34-4851-bc85-b30d170bf5b5\" (UID: \"fe95fb9a-7b34-4851-bc85-b30d170bf5b5\") "
Oct 30 00:53:35.300142 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:35.299656    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-openshift-service-ca\") pod \"fe95fb9a-7b34-4851-bc85-b30d170bf5b5\" (UID: \"fe95fb9a-7b34-4851-bc85-b30d170bf5b5\") "
Oct 30 00:53:35.300142 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:35.299810    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-metrics-certs\") pod \"fe95fb9a-7b34-4851-bc85-b30d170bf5b5\" (UID: \"fe95fb9a-7b34-4851-bc85-b30d170bf5b5\") "
Oct 30 00:53:35.300142 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:35.300119    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-openshift-service-ca" pod "fe95fb9a-7b34-4851-bc85-b30d170bf5b5" (UID: "fe95fb9a-7b34-4851-bc85-b30d170bf5b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:53:35.301892 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:35.301865    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-kube-api-access-tp5wk" pod "fe95fb9a-7b34-4851-bc85-b30d170bf5b5" (UID: "fe95fb9a-7b34-4851-bc85-b30d170bf5b5"). InnerVolumeSpecName "kube-api-access-tp5wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:53:35.302324 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:35.302301    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-maas-api-tls" pod "fe95fb9a-7b34-4851-bc85-b30d170bf5b5" (UID: "fe95fb9a-7b34-4851-bc85-b30d170bf5b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:53:35.302404 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:35.302318    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-metrics-certs" pod "fe95fb9a-7b34-4851-bc85-b30d170bf5b5" (UID: "fe95fb9a-7b34-4851-bc85-b30d170bf5b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:53:35.400781 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:35.400743    4864 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-metrics-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:35.400781 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:35.400773    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-maas-api-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:35.400781 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:35.400783    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp5wk\" (UniqueName: \"kubernetes.io/projected/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-kube-api-access-tp5wk\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:35.401003 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:35.400793    4864 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe95fb9a-7b34-4851-bc85-b30d170bf5b5-openshift-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:36.105432 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:36.105355    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s"
Oct 30 00:53:36.105432 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:36.105367    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s" event={"ID":"fe95fb9a-7b34-4851-bc85-b30d170bf5b5","Type":"ContainerDied","Data":"86a2185190bc63d027de27c11c4d7ddefbaa4005c51fad8946de6974833c5e28"}
Oct 30 00:53:36.105626 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:36.105453    4864 scope.go:122] "RemoveContainer" containerID="62cff40db10e7f79d253327a04d414b128865fa8ca7db0df9670f45889be1e10"
Oct 30 00:53:36.153586 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:36.153517    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s"]
Oct 30 00:53:36.163668 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:36.163637    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bbc42836-748599d5cc-fwf9s"]
Oct 30 00:53:36.581985 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:36.581949    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs"]
Oct 30 00:53:36.612227 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:36.612192    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7"]
Oct 30 00:53:37.371644 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:37.371603    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj"]
Oct 30 00:53:37.459898 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:37.459861    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw"]
Oct 30 00:53:37.460116 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:37.460084    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw" podUID="53924caf-da60-4fc7-8401-65d763f0a480" containerName="maas-api" containerID="cri-o://8e2db9c4b97ab1b55b6b407c08792d793e02284e630d2a9e66292ead4c4719f0" gracePeriod=30
Oct 30 00:53:37.722121 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:37.722094    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw"
Oct 30 00:53:37.793466 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:37.793430    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe95fb9a-7b34-4851-bc85-b30d170bf5b5" path="/var/lib/kubelet/pods/fe95fb9a-7b34-4851-bc85-b30d170bf5b5/volumes"
Oct 30 00:53:37.825347 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:37.825314    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53924caf-da60-4fc7-8401-65d763f0a480-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53924caf-da60-4fc7-8401-65d763f0a480-maas-api-tls\") pod \"53924caf-da60-4fc7-8401-65d763f0a480\" (UID: \"53924caf-da60-4fc7-8401-65d763f0a480\") "
Oct 30 00:53:37.825531 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:37.825405    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53924caf-da60-4fc7-8401-65d763f0a480-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53924caf-da60-4fc7-8401-65d763f0a480-openshift-service-ca\") pod \"53924caf-da60-4fc7-8401-65d763f0a480\" (UID: \"53924caf-da60-4fc7-8401-65d763f0a480\") "
Oct 30 00:53:37.825531 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:37.825485    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53924caf-da60-4fc7-8401-65d763f0a480-kube-api-access-x9cqs\" (UniqueName: \"kubernetes.io/projected/53924caf-da60-4fc7-8401-65d763f0a480-kube-api-access-x9cqs\") pod \"53924caf-da60-4fc7-8401-65d763f0a480\" (UID: \"53924caf-da60-4fc7-8401-65d763f0a480\") "
Oct 30 00:53:37.825531 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:37.825510    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53924caf-da60-4fc7-8401-65d763f0a480-metrics-certs\" (UniqueName: \"kubernetes.io/secret/53924caf-da60-4fc7-8401-65d763f0a480-metrics-certs\") pod \"53924caf-da60-4fc7-8401-65d763f0a480\" (UID: \"53924caf-da60-4fc7-8401-65d763f0a480\") "
Oct 30 00:53:37.825768 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:37.825744    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53924caf-da60-4fc7-8401-65d763f0a480-openshift-service-ca" pod "53924caf-da60-4fc7-8401-65d763f0a480" (UID: "53924caf-da60-4fc7-8401-65d763f0a480"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:53:37.825851 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:37.825833    4864 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53924caf-da60-4fc7-8401-65d763f0a480-openshift-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:37.827530 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:37.827499    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53924caf-da60-4fc7-8401-65d763f0a480-maas-api-tls" pod "53924caf-da60-4fc7-8401-65d763f0a480" (UID: "53924caf-da60-4fc7-8401-65d763f0a480"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:53:37.827641 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:37.827537    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53924caf-da60-4fc7-8401-65d763f0a480-metrics-certs" pod "53924caf-da60-4fc7-8401-65d763f0a480" (UID: "53924caf-da60-4fc7-8401-65d763f0a480"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:53:37.827641 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:37.827561    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53924caf-da60-4fc7-8401-65d763f0a480-kube-api-access-x9cqs" pod "53924caf-da60-4fc7-8401-65d763f0a480" (UID: "53924caf-da60-4fc7-8401-65d763f0a480"). InnerVolumeSpecName "kube-api-access-x9cqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:53:37.927325 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:37.927243    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9cqs\" (UniqueName: \"kubernetes.io/projected/53924caf-da60-4fc7-8401-65d763f0a480-kube-api-access-x9cqs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:37.927325 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:37.927275    4864 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53924caf-da60-4fc7-8401-65d763f0a480-metrics-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:37.927325 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:37.927284    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53924caf-da60-4fc7-8401-65d763f0a480-maas-api-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:37.966843 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:37.966809    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q"]
Oct 30 00:53:37.967015 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:37.966994    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q" podUID="564bc142-7b17-4f9f-bc53-b01980123954" containerName="maas-api" containerID="cri-o://a2b23601cac3a3b718878a8f5bcf2a6a22c0baada8221c791b21b9f327b67efd" gracePeriod=30
Oct 30 00:53:38.120756 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.120721    4864 generic.go:356] "Generic (PLEG): container finished" podID="53924caf-da60-4fc7-8401-65d763f0a480" containerID="8e2db9c4b97ab1b55b6b407c08792d793e02284e630d2a9e66292ead4c4719f0" exitCode=0
Oct 30 00:53:38.120943 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.120787    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw"
Oct 30 00:53:38.120943 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.120841    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw" event={"ID":"53924caf-da60-4fc7-8401-65d763f0a480","Type":"ContainerDied","Data":"8e2db9c4b97ab1b55b6b407c08792d793e02284e630d2a9e66292ead4c4719f0"}
Oct 30 00:53:38.120943 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.120924    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw" event={"ID":"53924caf-da60-4fc7-8401-65d763f0a480","Type":"ContainerDied","Data":"a527acdfff80427b70b63a43e954496c0afa513c320b98c8d60c1f67ea08ac3d"}
Oct 30 00:53:38.121108 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.120961    4864 scope.go:122] "RemoveContainer" containerID="8e2db9c4b97ab1b55b6b407c08792d793e02284e630d2a9e66292ead4c4719f0"
Oct 30 00:53:38.123303 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.123279    4864 generic.go:356] "Generic (PLEG): container finished" podID="564bc142-7b17-4f9f-bc53-b01980123954" containerID="a2b23601cac3a3b718878a8f5bcf2a6a22c0baada8221c791b21b9f327b67efd" exitCode=0
Oct 30 00:53:38.123303 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.123319    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q" event={"ID":"564bc142-7b17-4f9f-bc53-b01980123954","Type":"ContainerDied","Data":"a2b23601cac3a3b718878a8f5bcf2a6a22c0baada8221c791b21b9f327b67efd"}
Oct 30 00:53:38.132804 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.132772    4864 scope.go:122] "RemoveContainer" containerID="8e2db9c4b97ab1b55b6b407c08792d793e02284e630d2a9e66292ead4c4719f0"
Oct 30 00:53:38.133157 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:53:38.133129    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2db9c4b97ab1b55b6b407c08792d793e02284e630d2a9e66292ead4c4719f0\": container with ID starting with 8e2db9c4b97ab1b55b6b407c08792d793e02284e630d2a9e66292ead4c4719f0 not found: ID does not exist" containerID="8e2db9c4b97ab1b55b6b407c08792d793e02284e630d2a9e66292ead4c4719f0"
Oct 30 00:53:38.133297 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.133167    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2db9c4b97ab1b55b6b407c08792d793e02284e630d2a9e66292ead4c4719f0"} err="failed to get container status \"8e2db9c4b97ab1b55b6b407c08792d793e02284e630d2a9e66292ead4c4719f0\": rpc error: code = NotFound desc = could not find container \"8e2db9c4b97ab1b55b6b407c08792d793e02284e630d2a9e66292ead4c4719f0\": container with ID starting with 8e2db9c4b97ab1b55b6b407c08792d793e02284e630d2a9e66292ead4c4719f0 not found: ID does not exist"
Oct 30 00:53:38.179612 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.179531    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw"]
Oct 30 00:53:38.189426 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.189397    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-aaf98f1a-54d9f67b4c-9phpw"]
Oct 30 00:53:38.213107 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.213082    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q"
Oct 30 00:53:38.330189 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.330153    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/564bc142-7b17-4f9f-bc53-b01980123954-kube-api-access-gfw57\" (UniqueName: \"kubernetes.io/projected/564bc142-7b17-4f9f-bc53-b01980123954-kube-api-access-gfw57\") pod \"564bc142-7b17-4f9f-bc53-b01980123954\" (UID: \"564bc142-7b17-4f9f-bc53-b01980123954\") "
Oct 30 00:53:38.330389 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.330243    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/564bc142-7b17-4f9f-bc53-b01980123954-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/564bc142-7b17-4f9f-bc53-b01980123954-maas-api-tls\") pod \"564bc142-7b17-4f9f-bc53-b01980123954\" (UID: \"564bc142-7b17-4f9f-bc53-b01980123954\") "
Oct 30 00:53:38.330473 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.330451    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/564bc142-7b17-4f9f-bc53-b01980123954-metrics-certs\" (UniqueName: \"kubernetes.io/secret/564bc142-7b17-4f9f-bc53-b01980123954-metrics-certs\") pod \"564bc142-7b17-4f9f-bc53-b01980123954\" (UID: \"564bc142-7b17-4f9f-bc53-b01980123954\") "
Oct 30 00:53:38.330590 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.330570    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/564bc142-7b17-4f9f-bc53-b01980123954-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/564bc142-7b17-4f9f-bc53-b01980123954-openshift-service-ca\") pod \"564bc142-7b17-4f9f-bc53-b01980123954\" (UID: \"564bc142-7b17-4f9f-bc53-b01980123954\") "
Oct 30 00:53:38.330980 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.330948    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/564bc142-7b17-4f9f-bc53-b01980123954-openshift-service-ca" pod "564bc142-7b17-4f9f-bc53-b01980123954" (UID: "564bc142-7b17-4f9f-bc53-b01980123954"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:53:38.332255 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.332231    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/564bc142-7b17-4f9f-bc53-b01980123954-kube-api-access-gfw57" pod "564bc142-7b17-4f9f-bc53-b01980123954" (UID: "564bc142-7b17-4f9f-bc53-b01980123954"). InnerVolumeSpecName "kube-api-access-gfw57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:53:38.332400 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.332377    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/564bc142-7b17-4f9f-bc53-b01980123954-maas-api-tls" pod "564bc142-7b17-4f9f-bc53-b01980123954" (UID: "564bc142-7b17-4f9f-bc53-b01980123954"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:53:38.332471 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.332448    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/564bc142-7b17-4f9f-bc53-b01980123954-metrics-certs" pod "564bc142-7b17-4f9f-bc53-b01980123954" (UID: "564bc142-7b17-4f9f-bc53-b01980123954"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:53:38.431629 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.431539    4864 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/564bc142-7b17-4f9f-bc53-b01980123954-metrics-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:38.431629 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.431569    4864 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/564bc142-7b17-4f9f-bc53-b01980123954-openshift-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:38.431629 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.431580    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfw57\" (UniqueName: \"kubernetes.io/projected/564bc142-7b17-4f9f-bc53-b01980123954-kube-api-access-gfw57\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:38.431629 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.431590    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/564bc142-7b17-4f9f-bc53-b01980123954-maas-api-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:38.512365 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.512317    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx"]
Oct 30 00:53:38.512680 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.512653    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx" podUID="40d05fec-21fd-4c3e-9cce-f63766dbd08c" containerName="maas-api" containerID="cri-o://aa18d2bea6ae6230d69d84bde6ba1804442116d85e1e1ef952cb13da6aba5b9e" gracePeriod=30
Oct 30 00:53:38.585236 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.585203    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"]
Oct 30 00:53:38.585434 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.585410    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj" podUID="89c3fcfa-204a-43fc-b0fc-74658fd41ceb" containerName="istio-proxy" containerID="cri-o://4be419010029f4635d7334a2f93f0c6d2f4853843d0a114c26355fab89938c4c" gracePeriod=30
Oct 30 00:53:38.769282 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.769257    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx"
Oct 30 00:53:38.936700 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.936608    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d05fec-21fd-4c3e-9cce-f63766dbd08c-kube-api-access-v9h7c\" (UniqueName: \"kubernetes.io/projected/40d05fec-21fd-4c3e-9cce-f63766dbd08c-kube-api-access-v9h7c\") pod \"40d05fec-21fd-4c3e-9cce-f63766dbd08c\" (UID: \"40d05fec-21fd-4c3e-9cce-f63766dbd08c\") "
Oct 30 00:53:38.936700 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.936670    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40d05fec-21fd-4c3e-9cce-f63766dbd08c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/40d05fec-21fd-4c3e-9cce-f63766dbd08c-metrics-certs\") pod \"40d05fec-21fd-4c3e-9cce-f63766dbd08c\" (UID: \"40d05fec-21fd-4c3e-9cce-f63766dbd08c\") "
Oct 30 00:53:38.936937 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.936709    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40d05fec-21fd-4c3e-9cce-f63766dbd08c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40d05fec-21fd-4c3e-9cce-f63766dbd08c-maas-api-tls\") pod \"40d05fec-21fd-4c3e-9cce-f63766dbd08c\" (UID: \"40d05fec-21fd-4c3e-9cce-f63766dbd08c\") "
Oct 30 00:53:38.936937 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.936770    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40d05fec-21fd-4c3e-9cce-f63766dbd08c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d05fec-21fd-4c3e-9cce-f63766dbd08c-openshift-service-ca\") pod \"40d05fec-21fd-4c3e-9cce-f63766dbd08c\" (UID: \"40d05fec-21fd-4c3e-9cce-f63766dbd08c\") "
Oct 30 00:53:38.937277 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.937233    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d05fec-21fd-4c3e-9cce-f63766dbd08c-openshift-service-ca" pod "40d05fec-21fd-4c3e-9cce-f63766dbd08c" (UID: "40d05fec-21fd-4c3e-9cce-f63766dbd08c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:53:38.938811 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.938785    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40d05fec-21fd-4c3e-9cce-f63766dbd08c-metrics-certs" pod "40d05fec-21fd-4c3e-9cce-f63766dbd08c" (UID: "40d05fec-21fd-4c3e-9cce-f63766dbd08c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:53:38.939314 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.939291    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40d05fec-21fd-4c3e-9cce-f63766dbd08c-maas-api-tls" pod "40d05fec-21fd-4c3e-9cce-f63766dbd08c" (UID: "40d05fec-21fd-4c3e-9cce-f63766dbd08c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:53:38.940176 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:38.940156    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d05fec-21fd-4c3e-9cce-f63766dbd08c-kube-api-access-v9h7c" pod "40d05fec-21fd-4c3e-9cce-f63766dbd08c" (UID: "40d05fec-21fd-4c3e-9cce-f63766dbd08c"). InnerVolumeSpecName "kube-api-access-v9h7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:53:39.038217 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:39.038180    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9h7c\" (UniqueName: \"kubernetes.io/projected/40d05fec-21fd-4c3e-9cce-f63766dbd08c-kube-api-access-v9h7c\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:39.038217 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:39.038210    4864 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40d05fec-21fd-4c3e-9cce-f63766dbd08c-metrics-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:39.038217 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:39.038231    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40d05fec-21fd-4c3e-9cce-f63766dbd08c-maas-api-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:39.038454 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:39.038242    4864 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d05fec-21fd-4c3e-9cce-f63766dbd08c-openshift-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:39.129412 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:39.129376    4864 generic.go:356] "Generic (PLEG): container finished" podID="40d05fec-21fd-4c3e-9cce-f63766dbd08c" containerID="aa18d2bea6ae6230d69d84bde6ba1804442116d85e1e1ef952cb13da6aba5b9e" exitCode=0
Oct 30 00:53:39.129574 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:39.129423    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx"
Oct 30 00:53:39.129574 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:39.129463    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx" event={"ID":"40d05fec-21fd-4c3e-9cce-f63766dbd08c","Type":"ContainerDied","Data":"aa18d2bea6ae6230d69d84bde6ba1804442116d85e1e1ef952cb13da6aba5b9e"}
Oct 30 00:53:39.129574 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:39.129516    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx" event={"ID":"40d05fec-21fd-4c3e-9cce-f63766dbd08c","Type":"ContainerDied","Data":"3c6201270b6495312319a46cd181ac3567ba59bbc3cc3c2b8536798df114c826"}
Oct 30 00:53:39.129574 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:39.129546    4864 scope.go:122] "RemoveContainer" containerID="aa18d2bea6ae6230d69d84bde6ba1804442116d85e1e1ef952cb13da6aba5b9e"
Oct 30 00:53:39.131940 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:39.131921    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q"
Oct 30 00:53:39.132058 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:39.131921    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q" event={"ID":"564bc142-7b17-4f9f-bc53-b01980123954","Type":"ContainerDied","Data":"45fc849a169373072a0085a10c6a8ae72da977e2199921c88211bd48d6eebea5"}
Oct 30 00:53:39.139181 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:39.139158    4864 scope.go:122] "RemoveContainer" containerID="aa18d2bea6ae6230d69d84bde6ba1804442116d85e1e1ef952cb13da6aba5b9e"
Oct 30 00:53:39.139457 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:53:39.139438    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa18d2bea6ae6230d69d84bde6ba1804442116d85e1e1ef952cb13da6aba5b9e\": container with ID starting with aa18d2bea6ae6230d69d84bde6ba1804442116d85e1e1ef952cb13da6aba5b9e not found: ID does not exist" containerID="aa18d2bea6ae6230d69d84bde6ba1804442116d85e1e1ef952cb13da6aba5b9e"
Oct 30 00:53:39.139549 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:39.139464    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa18d2bea6ae6230d69d84bde6ba1804442116d85e1e1ef952cb13da6aba5b9e"} err="failed to get container status \"aa18d2bea6ae6230d69d84bde6ba1804442116d85e1e1ef952cb13da6aba5b9e\": rpc error: code = NotFound desc = could not find container \"aa18d2bea6ae6230d69d84bde6ba1804442116d85e1e1ef952cb13da6aba5b9e\": container with ID starting with aa18d2bea6ae6230d69d84bde6ba1804442116d85e1e1ef952cb13da6aba5b9e not found: ID does not exist"
Oct 30 00:53:39.139549 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:39.139484    4864 scope.go:122] "RemoveContainer" containerID="a2b23601cac3a3b718878a8f5bcf2a6a22c0baada8221c791b21b9f327b67efd"
Oct 30 00:53:39.178670 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:39.178623    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx"]
Oct 30 00:53:39.192552 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:39.192482    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b06528d1-7989bc84fb-m68zx"]
Oct 30 00:53:39.251451 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:39.251425    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q"]
Oct 30 00:53:39.274206 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:39.274177    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3cc211d9-55b788c775-khc7q"]
Oct 30 00:53:39.792852 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:39.792822    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40d05fec-21fd-4c3e-9cce-f63766dbd08c" path="/var/lib/kubelet/pods/40d05fec-21fd-4c3e-9cce-f63766dbd08c/volumes"
Oct 30 00:53:39.793248 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:39.793234    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53924caf-da60-4fc7-8401-65d763f0a480" path="/var/lib/kubelet/pods/53924caf-da60-4fc7-8401-65d763f0a480/volumes"
Oct 30 00:53:39.793590 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:39.793578    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="564bc142-7b17-4f9f-bc53-b01980123954" path="/var/lib/kubelet/pods/564bc142-7b17-4f9f-bc53-b01980123954/volumes"
Oct 30 00:53:40.032530 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.032499    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"]
Oct 30 00:53:40.032940 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.032922    4864 cpu_manager.go:414] "containerMap: removing container" podUID="fe95fb9a-7b34-4851-bc85-b30d170bf5b5" containerName="maas-api"
Oct 30 00:53:40.032940 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.032936    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe95fb9a-7b34-4851-bc85-b30d170bf5b5" containerName="maas-api"
Oct 30 00:53:40.032940 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.032947    4864 cpu_manager.go:414] "containerMap: removing container" podUID="53924caf-da60-4fc7-8401-65d763f0a480" containerName="maas-api"
Oct 30 00:53:40.033212 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.032952    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53924caf-da60-4fc7-8401-65d763f0a480" containerName="maas-api"
Oct 30 00:53:40.033212 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.032960    4864 cpu_manager.go:414] "containerMap: removing container" podUID="a26d9d81-1722-4d9f-97c8-f4065502affc" containerName="payload-pre-processing"
Oct 30 00:53:40.033212 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.032965    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a26d9d81-1722-4d9f-97c8-f4065502affc" containerName="payload-pre-processing"
Oct 30 00:53:40.033212 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.032979    4864 cpu_manager.go:414] "containerMap: removing container" podUID="564bc142-7b17-4f9f-bc53-b01980123954" containerName="maas-api"
Oct 30 00:53:40.033212 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.032983    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="564bc142-7b17-4f9f-bc53-b01980123954" containerName="maas-api"
Oct 30 00:53:40.033212 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.033001    4864 cpu_manager.go:414] "containerMap: removing container" podUID="40d05fec-21fd-4c3e-9cce-f63766dbd08c" containerName="maas-api"
Oct 30 00:53:40.033212 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.033005    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d05fec-21fd-4c3e-9cce-f63766dbd08c" containerName="maas-api"
Oct 30 00:53:40.033212 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.033111    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="564bc142-7b17-4f9f-bc53-b01980123954" containerName="maas-api"
Oct 30 00:53:40.033212 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.033124    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d05fec-21fd-4c3e-9cce-f63766dbd08c" containerName="maas-api"
Oct 30 00:53:40.033212 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.033133    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe95fb9a-7b34-4851-bc85-b30d170bf5b5" containerName="maas-api"
Oct 30 00:53:40.033212 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.033139    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="53924caf-da60-4fc7-8401-65d763f0a480" containerName="maas-api"
Oct 30 00:53:40.033212 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.033144    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="a26d9d81-1722-4d9f-97c8-f4065502affc" containerName="payload-pre-processing"
Oct 30 00:53:40.037710 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.037686    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.061288 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.061264    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-664c2b-openshift-default-dockercfg-vvttz\""
Oct 30 00:53:40.084104 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.084062    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"]
Oct 30 00:53:40.148690 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.148655    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08db946-5b81-477f-bc5a-8537b0385100-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.148857 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.148712    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcld5\" (UniqueName: \"kubernetes.io/projected/c08db946-5b81-477f-bc5a-8537b0385100-kube-api-access-kcld5\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.148857 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.148805    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.148857 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.148848    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.148972 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.148903    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.148972 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.148939    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.148972 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.148969    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.149093 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.148992    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08db946-5b81-477f-bc5a-8537b0385100-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.149093 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.149054    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08db946-5b81-477f-bc5a-8537b0385100-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.249981 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.249939    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.250187 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.249996    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08db946-5b81-477f-bc5a-8537b0385100-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.250187 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.250025    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08db946-5b81-477f-bc5a-8537b0385100-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.250187 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.250066    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08db946-5b81-477f-bc5a-8537b0385100-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.250187 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.250167    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcld5\" (UniqueName: \"kubernetes.io/projected/c08db946-5b81-477f-bc5a-8537b0385100-kube-api-access-kcld5\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.250447 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.250209    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.250447 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.250268    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.250447 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.250304    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.250447 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.250355    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.250792 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.250768    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.251117 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.250952    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.251117 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.251041    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.251262 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.251108    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08db946-5b81-477f-bc5a-8537b0385100-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.251629 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.251601    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.252934 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.252912    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08db946-5b81-477f-bc5a-8537b0385100-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.253021 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.253002    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.271768 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.271743    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08db946-5b81-477f-bc5a-8537b0385100-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.272567 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.272542    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcld5\" (UniqueName: \"kubernetes.io/projected/c08db946-5b81-477f-bc5a-8537b0385100-kube-api-access-kcld5\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.351306 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.351234    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:40.687816 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.687727    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"]
Oct 30 00:53:40.691857 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:53:40.691827    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08db946_5b81_477f_bc5a_8537b0385100.slice/crio-8a6d15f8d2f961c326d35650e88457d49a64b562584c02a635616cddbd96422d WatchSource:0}: Error finding container 8a6d15f8d2f961c326d35650e88457d49a64b562584c02a635616cddbd96422d: Status 404 returned error can't find the container with id 8a6d15f8d2f961c326d35650e88457d49a64b562584c02a635616cddbd96422d
Oct 30 00:53:40.693905 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.693877    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:53:40.693961 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.693939    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:53:40.694000 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:40.693970    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:53:41.146386 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:41.146346    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c" event={"ID":"c08db946-5b81-477f-bc5a-8537b0385100","Type":"ContainerStarted","Data":"026f8125a099382faf1e5739df3b1d06eb4ecec768b5049395b8d1f2edc8fd4a"}
Oct 30 00:53:41.146857 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:41.146438    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c" event={"ID":"c08db946-5b81-477f-bc5a-8537b0385100","Type":"ContainerStarted","Data":"8a6d15f8d2f961c326d35650e88457d49a64b562584c02a635616cddbd96422d"}
Oct 30 00:53:41.351582 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:41.351549    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:41.356207 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:41.356183    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:41.385770 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:41.385740    4864 patch_prober.go:29] interesting pod/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 00:53:41.385889 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:41.385790    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj" podUID="89c3fcfa-204a-43fc-b0fc-74658fd41ceb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 00:53:41.403032 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:41.402940    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c" podStartSLOduration=2.402927798 podStartE2EDuration="2.402927798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:41.21861112 +0000 UTC m=+733.556713049" watchObservedRunningTime="2026-10-30 00:53:41.402927798 +0000 UTC m=+733.741029707"
Oct 30 00:53:41.582481 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:41.582434    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs" podUID="ac105c72-22ed-456a-807b-90f4d672da14" containerName="payload-processing" containerID="cri-o://82cfa5c1482962d531bb4cf3c035844385fd1fc5cb91f8ef65845b0c6b813fe5" gracePeriod=25
Oct 30 00:53:41.612892 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:41.612859    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7" podUID="481bc47b-b2f6-4d18-a747-413ac025eebb" containerName="payload-pre-processing" containerID="cri-o://fb6f3da2c98f37a617a485290fbcbcb9a3410a099cc64296fa96ba55e25b5f0a" gracePeriod=25
Oct 30 00:53:41.864711 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:41.864687    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs"
Oct 30 00:53:41.888633 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:41.888612    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7"
Oct 30 00:53:41.967912 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:41.967871    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac105c72-22ed-456a-807b-90f4d672da14-kube-api-access-2fvwl\" (UniqueName: \"kubernetes.io/projected/ac105c72-22ed-456a-807b-90f4d672da14-kube-api-access-2fvwl\") pod \"ac105c72-22ed-456a-807b-90f4d672da14\" (UID: \"ac105c72-22ed-456a-807b-90f4d672da14\") "
Oct 30 00:53:41.968086 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:41.967979    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac105c72-22ed-456a-807b-90f4d672da14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac105c72-22ed-456a-807b-90f4d672da14-plugins-config-volume\") pod \"ac105c72-22ed-456a-807b-90f4d672da14\" (UID: \"ac105c72-22ed-456a-807b-90f4d672da14\") "
Oct 30 00:53:41.968442 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:41.968368    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac105c72-22ed-456a-807b-90f4d672da14-plugins-config-volume" pod "ac105c72-22ed-456a-807b-90f4d672da14" (UID: "ac105c72-22ed-456a-807b-90f4d672da14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:53:41.970108 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:41.970064    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac105c72-22ed-456a-807b-90f4d672da14-kube-api-access-2fvwl" pod "ac105c72-22ed-456a-807b-90f4d672da14" (UID: "ac105c72-22ed-456a-807b-90f4d672da14"). InnerVolumeSpecName "kube-api-access-2fvwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:53:42.069180 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.069142    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/481bc47b-b2f6-4d18-a747-413ac025eebb-kube-api-access-wmzc6\" (UniqueName: \"kubernetes.io/projected/481bc47b-b2f6-4d18-a747-413ac025eebb-kube-api-access-wmzc6\") pod \"481bc47b-b2f6-4d18-a747-413ac025eebb\" (UID: \"481bc47b-b2f6-4d18-a747-413ac025eebb\") "
Oct 30 00:53:42.069327 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.069262    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/481bc47b-b2f6-4d18-a747-413ac025eebb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/481bc47b-b2f6-4d18-a747-413ac025eebb-plugins-config-volume\") pod \"481bc47b-b2f6-4d18-a747-413ac025eebb\" (UID: \"481bc47b-b2f6-4d18-a747-413ac025eebb\") "
Oct 30 00:53:42.069496 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.069477    4864 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac105c72-22ed-456a-807b-90f4d672da14-plugins-config-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:42.069582 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.069498    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fvwl\" (UniqueName: \"kubernetes.io/projected/ac105c72-22ed-456a-807b-90f4d672da14-kube-api-access-2fvwl\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:42.069671 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.069647    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481bc47b-b2f6-4d18-a747-413ac025eebb-plugins-config-volume" pod "481bc47b-b2f6-4d18-a747-413ac025eebb" (UID: "481bc47b-b2f6-4d18-a747-413ac025eebb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:53:42.071124 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.071105    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481bc47b-b2f6-4d18-a747-413ac025eebb-kube-api-access-wmzc6" pod "481bc47b-b2f6-4d18-a747-413ac025eebb" (UID: "481bc47b-b2f6-4d18-a747-413ac025eebb"). InnerVolumeSpecName "kube-api-access-wmzc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:53:42.152723 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.152686    4864 generic.go:356] "Generic (PLEG): container finished" podID="481bc47b-b2f6-4d18-a747-413ac025eebb" containerID="fb6f3da2c98f37a617a485290fbcbcb9a3410a099cc64296fa96ba55e25b5f0a" exitCode=0
Oct 30 00:53:42.153202 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.152738    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7"
Oct 30 00:53:42.153202 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.152773    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7" event={"ID":"481bc47b-b2f6-4d18-a747-413ac025eebb","Type":"ContainerDied","Data":"fb6f3da2c98f37a617a485290fbcbcb9a3410a099cc64296fa96ba55e25b5f0a"}
Oct 30 00:53:42.153202 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.152827    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7" event={"ID":"481bc47b-b2f6-4d18-a747-413ac025eebb","Type":"ContainerDied","Data":"5661a204f40f90329d719a7d60d63789393403ba672359b674f2acc1032a06d3"}
Oct 30 00:53:42.153202 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.152855    4864 scope.go:122] "RemoveContainer" containerID="fb6f3da2c98f37a617a485290fbcbcb9a3410a099cc64296fa96ba55e25b5f0a"
Oct 30 00:53:42.154214 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.154195    4864 generic.go:356] "Generic (PLEG): container finished" podID="ac105c72-22ed-456a-807b-90f4d672da14" containerID="82cfa5c1482962d531bb4cf3c035844385fd1fc5cb91f8ef65845b0c6b813fe5" exitCode=0
Oct 30 00:53:42.154296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.154254    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs"
Oct 30 00:53:42.154296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.154259    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs" event={"ID":"ac105c72-22ed-456a-807b-90f4d672da14","Type":"ContainerDied","Data":"82cfa5c1482962d531bb4cf3c035844385fd1fc5cb91f8ef65845b0c6b813fe5"}
Oct 30 00:53:42.154389 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.154305    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs" event={"ID":"ac105c72-22ed-456a-807b-90f4d672da14","Type":"ContainerDied","Data":"19216143d7ffa30eb28c43c911768107f74ddbd438817a0db9e0900b25cf67db"}
Oct 30 00:53:42.154579 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.154556    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:42.155665 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.155643    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:53:42.163236 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.163218    4864 scope.go:122] "RemoveContainer" containerID="fb6f3da2c98f37a617a485290fbcbcb9a3410a099cc64296fa96ba55e25b5f0a"
Oct 30 00:53:42.163499 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:53:42.163479    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb6f3da2c98f37a617a485290fbcbcb9a3410a099cc64296fa96ba55e25b5f0a\": container with ID starting with fb6f3da2c98f37a617a485290fbcbcb9a3410a099cc64296fa96ba55e25b5f0a not found: ID does not exist" containerID="fb6f3da2c98f37a617a485290fbcbcb9a3410a099cc64296fa96ba55e25b5f0a"
Oct 30 00:53:42.163499 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.163504    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6f3da2c98f37a617a485290fbcbcb9a3410a099cc64296fa96ba55e25b5f0a"} err="failed to get container status \"fb6f3da2c98f37a617a485290fbcbcb9a3410a099cc64296fa96ba55e25b5f0a\": rpc error: code = NotFound desc = could not find container \"fb6f3da2c98f37a617a485290fbcbcb9a3410a099cc64296fa96ba55e25b5f0a\": container with ID starting with fb6f3da2c98f37a617a485290fbcbcb9a3410a099cc64296fa96ba55e25b5f0a not found: ID does not exist"
Oct 30 00:53:42.163499 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.163521    4864 scope.go:122] "RemoveContainer" containerID="82cfa5c1482962d531bb4cf3c035844385fd1fc5cb91f8ef65845b0c6b813fe5"
Oct 30 00:53:42.170566 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.170539    4864 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/481bc47b-b2f6-4d18-a747-413ac025eebb-plugins-config-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:42.170566 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.170559    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmzc6\" (UniqueName: \"kubernetes.io/projected/481bc47b-b2f6-4d18-a747-413ac025eebb-kube-api-access-wmzc6\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:42.171539 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.171518    4864 scope.go:122] "RemoveContainer" containerID="82cfa5c1482962d531bb4cf3c035844385fd1fc5cb91f8ef65845b0c6b813fe5"
Oct 30 00:53:42.171793 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:53:42.171779    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82cfa5c1482962d531bb4cf3c035844385fd1fc5cb91f8ef65845b0c6b813fe5\": container with ID starting with 82cfa5c1482962d531bb4cf3c035844385fd1fc5cb91f8ef65845b0c6b813fe5 not found: ID does not exist" containerID="82cfa5c1482962d531bb4cf3c035844385fd1fc5cb91f8ef65845b0c6b813fe5"
Oct 30 00:53:42.171847 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.171798    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82cfa5c1482962d531bb4cf3c035844385fd1fc5cb91f8ef65845b0c6b813fe5"} err="failed to get container status \"82cfa5c1482962d531bb4cf3c035844385fd1fc5cb91f8ef65845b0c6b813fe5\": rpc error: code = NotFound desc = could not find container \"82cfa5c1482962d531bb4cf3c035844385fd1fc5cb91f8ef65845b0c6b813fe5\": container with ID starting with 82cfa5c1482962d531bb4cf3c035844385fd1fc5cb91f8ef65845b0c6b813fe5 not found: ID does not exist"
Oct 30 00:53:42.273466 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.273373    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs"]
Oct 30 00:53:42.280051 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.280025    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b06528d1-74749cb6f6-p84rs"]
Oct 30 00:53:42.321664 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.321633    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7"]
Oct 30 00:53:42.326323 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.326298    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b06528d1-6665b966dc-6nfg7"]
Oct 30 00:53:42.372239 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.372206    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj" podUID="8588a088-9405-4acf-be5b-73aefb0c2402" containerName="payload-pre-processing" containerID="cri-o://00d82f64aaf7526273cd1c5e70a81d22fde3f7ba29d55591f1eeb8f89464f11a" gracePeriod=25
Oct 30 00:53:42.623804 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.623779    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj"
Oct 30 00:53:42.775834 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.775797    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8588a088-9405-4acf-be5b-73aefb0c2402-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8588a088-9405-4acf-be5b-73aefb0c2402-plugins-config-volume\") pod \"8588a088-9405-4acf-be5b-73aefb0c2402\" (UID: \"8588a088-9405-4acf-be5b-73aefb0c2402\") "
Oct 30 00:53:42.775976 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.775867    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8588a088-9405-4acf-be5b-73aefb0c2402-kube-api-access-s4g2m\" (UniqueName: \"kubernetes.io/projected/8588a088-9405-4acf-be5b-73aefb0c2402-kube-api-access-s4g2m\") pod \"8588a088-9405-4acf-be5b-73aefb0c2402\" (UID: \"8588a088-9405-4acf-be5b-73aefb0c2402\") "
Oct 30 00:53:42.776210 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.776188    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8588a088-9405-4acf-be5b-73aefb0c2402-plugins-config-volume" pod "8588a088-9405-4acf-be5b-73aefb0c2402" (UID: "8588a088-9405-4acf-be5b-73aefb0c2402"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:53:42.777853 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.777833    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8588a088-9405-4acf-be5b-73aefb0c2402-kube-api-access-s4g2m" pod "8588a088-9405-4acf-be5b-73aefb0c2402" (UID: "8588a088-9405-4acf-be5b-73aefb0c2402"). InnerVolumeSpecName "kube-api-access-s4g2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:53:42.876902 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.876831    4864 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8588a088-9405-4acf-be5b-73aefb0c2402-plugins-config-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:42.876902 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:42.876855    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4g2m\" (UniqueName: \"kubernetes.io/projected/8588a088-9405-4acf-be5b-73aefb0c2402-kube-api-access-s4g2m\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:43.162133 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.162031    4864 generic.go:356] "Generic (PLEG): container finished" podID="8588a088-9405-4acf-be5b-73aefb0c2402" containerID="00d82f64aaf7526273cd1c5e70a81d22fde3f7ba29d55591f1eeb8f89464f11a" exitCode=0
Oct 30 00:53:43.162133 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.162091    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj"
Oct 30 00:53:43.162133 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.162114    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj" event={"ID":"8588a088-9405-4acf-be5b-73aefb0c2402","Type":"ContainerDied","Data":"00d82f64aaf7526273cd1c5e70a81d22fde3f7ba29d55591f1eeb8f89464f11a"}
Oct 30 00:53:43.162692 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.162172    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj" event={"ID":"8588a088-9405-4acf-be5b-73aefb0c2402","Type":"ContainerDied","Data":"11a69a392203e67209d83827d0d27fe3ca2b80cb13e1c8177e386f7a4071fedb"}
Oct 30 00:53:43.162692 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.162198    4864 scope.go:122] "RemoveContainer" containerID="00d82f64aaf7526273cd1c5e70a81d22fde3f7ba29d55591f1eeb8f89464f11a"
Oct 30 00:53:43.171387 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.171368    4864 scope.go:122] "RemoveContainer" containerID="00d82f64aaf7526273cd1c5e70a81d22fde3f7ba29d55591f1eeb8f89464f11a"
Oct 30 00:53:43.171668 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:53:43.171648    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00d82f64aaf7526273cd1c5e70a81d22fde3f7ba29d55591f1eeb8f89464f11a\": container with ID starting with 00d82f64aaf7526273cd1c5e70a81d22fde3f7ba29d55591f1eeb8f89464f11a not found: ID does not exist" containerID="00d82f64aaf7526273cd1c5e70a81d22fde3f7ba29d55591f1eeb8f89464f11a"
Oct 30 00:53:43.171719 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.171676    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00d82f64aaf7526273cd1c5e70a81d22fde3f7ba29d55591f1eeb8f89464f11a"} err="failed to get container status \"00d82f64aaf7526273cd1c5e70a81d22fde3f7ba29d55591f1eeb8f89464f11a\": rpc error: code = NotFound desc = could not find container \"00d82f64aaf7526273cd1c5e70a81d22fde3f7ba29d55591f1eeb8f89464f11a\": container with ID starting with 00d82f64aaf7526273cd1c5e70a81d22fde3f7ba29d55591f1eeb8f89464f11a not found: ID does not exist"
Oct 30 00:53:43.208156 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.208127    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj"]
Oct 30 00:53:43.218261 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.218239    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3cc211d9-586869df75-25qzj"]
Oct 30 00:53:43.793210 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.793179    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="481bc47b-b2f6-4d18-a747-413ac025eebb" path="/var/lib/kubelet/pods/481bc47b-b2f6-4d18-a747-413ac025eebb/volumes"
Oct 30 00:53:43.793678 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.793661    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8588a088-9405-4acf-be5b-73aefb0c2402" path="/var/lib/kubelet/pods/8588a088-9405-4acf-be5b-73aefb0c2402/volumes"
Oct 30 00:53:43.794044 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.794032    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac105c72-22ed-456a-807b-90f4d672da14" path="/var/lib/kubelet/pods/ac105c72-22ed-456a-807b-90f4d672da14/volumes"
Oct 30 00:53:43.839599 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.839574    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:53:43.987570 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.987537    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-podinfo\") pod \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") "
Oct 30 00:53:43.987759 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.987581    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-kube-api-access-nq66q\" (UniqueName: \"kubernetes.io/projected/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-kube-api-access-nq66q\") pod \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") "
Oct 30 00:53:43.987759 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.987613    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-workload-socket\") pod \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") "
Oct 30 00:53:43.987759 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.987650    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-envoy\") pod \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") "
Oct 30 00:53:43.987759 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.987685    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-credential-socket\") pod \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") "
Oct 30 00:53:43.987759 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.987733    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istiod-ca-cert\") pod \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") "
Oct 30 00:53:43.988024 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.987776    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-workload-certs\") pod \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") "
Oct 30 00:53:43.988024 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.987822    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-data\") pod \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") "
Oct 30 00:53:43.988024 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.987898    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-token\" (UniqueName: \"kubernetes.io/projected/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-token\") pod \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\" (UID: \"89c3fcfa-204a-43fc-b0fc-74658fd41ceb\") "
Oct 30 00:53:43.988024 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.987962    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-workload-socket" pod "89c3fcfa-204a-43fc-b0fc-74658fd41ceb" (UID: "89c3fcfa-204a-43fc-b0fc-74658fd41ceb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:53:43.988024 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.988002    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-credential-socket" pod "89c3fcfa-204a-43fc-b0fc-74658fd41ceb" (UID: "89c3fcfa-204a-43fc-b0fc-74658fd41ceb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:53:43.988304 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.988217    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-data" pod "89c3fcfa-204a-43fc-b0fc-74658fd41ceb" (UID: "89c3fcfa-204a-43fc-b0fc-74658fd41ceb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:53:43.988304 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.988266    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istiod-ca-cert" pod "89c3fcfa-204a-43fc-b0fc-74658fd41ceb" (UID: "89c3fcfa-204a-43fc-b0fc-74658fd41ceb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:53:43.988400 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.988329    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:43.988400 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.988348    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:43.988400 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.988362    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:43.988536 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.988504    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-workload-certs" pod "89c3fcfa-204a-43fc-b0fc-74658fd41ceb" (UID: "89c3fcfa-204a-43fc-b0fc-74658fd41ceb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:53:43.989863 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.989837    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-envoy" pod "89c3fcfa-204a-43fc-b0fc-74658fd41ceb" (UID: "89c3fcfa-204a-43fc-b0fc-74658fd41ceb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:53:43.989971 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.989934    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-podinfo" pod "89c3fcfa-204a-43fc-b0fc-74658fd41ceb" (UID: "89c3fcfa-204a-43fc-b0fc-74658fd41ceb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:53:43.990261 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.990243    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-kube-api-access-nq66q" pod "89c3fcfa-204a-43fc-b0fc-74658fd41ceb" (UID: "89c3fcfa-204a-43fc-b0fc-74658fd41ceb"). InnerVolumeSpecName "kube-api-access-nq66q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:53:43.990261 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:43.990253    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-token" pod "89c3fcfa-204a-43fc-b0fc-74658fd41ceb" (UID: "89c3fcfa-204a-43fc-b0fc-74658fd41ceb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:53:44.089456 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:44.089426    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:44.089456 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:44.089458    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:44.089667 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:44.089472    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq66q\" (UniqueName: \"kubernetes.io/projected/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-kube-api-access-nq66q\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:44.089667 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:44.089486    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:44.089667 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:44.089495    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:44.089667 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:44.089503    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c3fcfa-204a-43fc-b0fc-74658fd41ceb-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:53:44.167874 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:44.167837    4864 generic.go:356] "Generic (PLEG): container finished" podID="89c3fcfa-204a-43fc-b0fc-74658fd41ceb" containerID="4be419010029f4635d7334a2f93f0c6d2f4853843d0a114c26355fab89938c4c" exitCode=0
Oct 30 00:53:44.168409 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:44.167873    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj" event={"ID":"89c3fcfa-204a-43fc-b0fc-74658fd41ceb","Type":"ContainerDied","Data":"4be419010029f4635d7334a2f93f0c6d2f4853843d0a114c26355fab89938c4c"}
Oct 30 00:53:44.168409 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:44.167931    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj" event={"ID":"89c3fcfa-204a-43fc-b0fc-74658fd41ceb","Type":"ContainerDied","Data":"9b4411f28d2d782e3788f9930129a00daaca138cc5c00b6716eaec1f2b42bb8f"}
Oct 30 00:53:44.168409 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:44.167886    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"
Oct 30 00:53:44.168409 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:44.167961    4864 scope.go:122] "RemoveContainer" containerID="4be419010029f4635d7334a2f93f0c6d2f4853843d0a114c26355fab89938c4c"
Oct 30 00:53:44.177683 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:44.177667    4864 scope.go:122] "RemoveContainer" containerID="31ad93f99084d1685591a97b39f2759348dabe2a3d66b0e09d425e8cf3da77d6"
Oct 30 00:53:44.185767 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:44.185747    4864 scope.go:122] "RemoveContainer" containerID="4be419010029f4635d7334a2f93f0c6d2f4853843d0a114c26355fab89938c4c"
Oct 30 00:53:44.185999 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:53:44.185984    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4be419010029f4635d7334a2f93f0c6d2f4853843d0a114c26355fab89938c4c\": container with ID starting with 4be419010029f4635d7334a2f93f0c6d2f4853843d0a114c26355fab89938c4c not found: ID does not exist" containerID="4be419010029f4635d7334a2f93f0c6d2f4853843d0a114c26355fab89938c4c"
Oct 30 00:53:44.186049 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:44.186007    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be419010029f4635d7334a2f93f0c6d2f4853843d0a114c26355fab89938c4c"} err="failed to get container status \"4be419010029f4635d7334a2f93f0c6d2f4853843d0a114c26355fab89938c4c\": rpc error: code = NotFound desc = could not find container \"4be419010029f4635d7334a2f93f0c6d2f4853843d0a114c26355fab89938c4c\": container with ID starting with 4be419010029f4635d7334a2f93f0c6d2f4853843d0a114c26355fab89938c4c not found: ID does not exist"
Oct 30 00:53:44.186049 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:44.186024    4864 scope.go:122] "RemoveContainer" containerID="31ad93f99084d1685591a97b39f2759348dabe2a3d66b0e09d425e8cf3da77d6"
Oct 30 00:53:44.186439 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:53:44.186418    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31ad93f99084d1685591a97b39f2759348dabe2a3d66b0e09d425e8cf3da77d6\": container with ID starting with 31ad93f99084d1685591a97b39f2759348dabe2a3d66b0e09d425e8cf3da77d6 not found: ID does not exist" containerID="31ad93f99084d1685591a97b39f2759348dabe2a3d66b0e09d425e8cf3da77d6"
Oct 30 00:53:44.186509 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:44.186443    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31ad93f99084d1685591a97b39f2759348dabe2a3d66b0e09d425e8cf3da77d6"} err="failed to get container status \"31ad93f99084d1685591a97b39f2759348dabe2a3d66b0e09d425e8cf3da77d6\": rpc error: code = NotFound desc = could not find container \"31ad93f99084d1685591a97b39f2759348dabe2a3d66b0e09d425e8cf3da77d6\": container with ID starting with 31ad93f99084d1685591a97b39f2759348dabe2a3d66b0e09d425e8cf3da77d6 not found: ID does not exist"
Oct 30 00:53:44.217484 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:44.217456    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"]
Oct 30 00:53:44.225139 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:44.225114    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-aaf98f1a-openshift-default-6cb4594789-mpqlj"]
Oct 30 00:53:45.791747 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:45.791714    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89c3fcfa-204a-43fc-b0fc-74658fd41ceb" path="/var/lib/kubelet/pods/89c3fcfa-204a-43fc-b0fc-74658fd41ceb/volumes"
Oct 30 00:53:49.038791 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:53:49.038755    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:53:50.912192 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.912133    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"]
Oct 30 00:53:50.912785 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.912764    4864 cpu_manager.go:414] "containerMap: removing container" podUID="89c3fcfa-204a-43fc-b0fc-74658fd41ceb" containerName="istio-proxy"
Oct 30 00:53:50.912876 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.912787    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89c3fcfa-204a-43fc-b0fc-74658fd41ceb" containerName="istio-proxy"
Oct 30 00:53:50.912876 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.912816    4864 cpu_manager.go:414] "containerMap: removing container" podUID="89c3fcfa-204a-43fc-b0fc-74658fd41ceb" containerName="istio-proxy"
Oct 30 00:53:50.912876 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.912824    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89c3fcfa-204a-43fc-b0fc-74658fd41ceb" containerName="istio-proxy"
Oct 30 00:53:50.912876 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.912844    4864 cpu_manager.go:414] "containerMap: removing container" podUID="481bc47b-b2f6-4d18-a747-413ac025eebb" containerName="payload-pre-processing"
Oct 30 00:53:50.912876 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.912853    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="481bc47b-b2f6-4d18-a747-413ac025eebb" containerName="payload-pre-processing"
Oct 30 00:53:50.912876 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.912866    4864 cpu_manager.go:414] "containerMap: removing container" podUID="8588a088-9405-4acf-be5b-73aefb0c2402" containerName="payload-pre-processing"
Oct 30 00:53:50.912876 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.912873    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8588a088-9405-4acf-be5b-73aefb0c2402" containerName="payload-pre-processing"
Oct 30 00:53:50.913242 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.912891    4864 cpu_manager.go:414] "containerMap: removing container" podUID="ac105c72-22ed-456a-807b-90f4d672da14" containerName="payload-processing"
Oct 30 00:53:50.913242 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.912899    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac105c72-22ed-456a-807b-90f4d672da14" containerName="payload-processing"
Oct 30 00:53:50.913242 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.913017    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="89c3fcfa-204a-43fc-b0fc-74658fd41ceb" containerName="istio-proxy"
Oct 30 00:53:50.913242 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.913033    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="89c3fcfa-204a-43fc-b0fc-74658fd41ceb" containerName="istio-proxy"
Oct 30 00:53:50.913242 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.913043    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="481bc47b-b2f6-4d18-a747-413ac025eebb" containerName="payload-pre-processing"
Oct 30 00:53:50.913242 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.913052    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="8588a088-9405-4acf-be5b-73aefb0c2402" containerName="payload-pre-processing"
Oct 30 00:53:50.913242 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.913063    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac105c72-22ed-456a-807b-90f4d672da14" containerName="payload-processing"
Oct 30 00:53:50.917833 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.917809    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:50.931882 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.931858    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-39b241-openshift-default-dockercfg-bchvq\""
Oct 30 00:53:50.946523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.946494    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:50.946645 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.946536    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:50.946645 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.946570    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:50.946743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.946644    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ae4638-c9e9-4d20-b699-feb3208662a2-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:50.946743 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.946707    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:50.946815 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.946799    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:50.946861 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.946827    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g998b\" (UniqueName: \"kubernetes.io/projected/e6ae4638-c9e9-4d20-b699-feb3208662a2-kube-api-access-g998b\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:50.947537 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.947511    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:50.947639 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.947552    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:50.948723 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:50.948702    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"]
Oct 30 00:53:51.048609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:51.048574    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:51.048779 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:51.048622    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:51.048779 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:51.048670    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:51.048779 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:51.048691    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:51.048779 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:51.048720    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:51.048936 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:51.048862    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ae4638-c9e9-4d20-b699-feb3208662a2-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:51.048986 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:51.048939    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:51.049052 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:51.049003    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:51.049052 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:51.049014    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:51.049052 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:51.049033    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g998b\" (UniqueName: \"kubernetes.io/projected/e6ae4638-c9e9-4d20-b699-feb3208662a2-kube-api-access-g998b\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:51.049271 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:51.049106    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:51.049337 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:51.049297    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:51.049393 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:51.049349    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:51.049660 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:51.049637    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ae4638-c9e9-4d20-b699-feb3208662a2-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:51.050926 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:51.050909    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:51.051271 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:51.051254    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:51.070688 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:51.070658    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:51.070787 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:51.070708    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g998b\" (UniqueName: \"kubernetes.io/projected/e6ae4638-c9e9-4d20-b699-feb3208662a2-kube-api-access-g998b\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:51.234910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:51.234873    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:51.546805 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:51.546713    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"]
Oct 30 00:53:51.551026 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:53:51.551001    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ae4638_c9e9_4d20_b699_feb3208662a2.slice/crio-b0b106acdd9445a13f75c0a44c656803e6d3644b36c12e5956b968634b692447 WatchSource:0}: Error finding container b0b106acdd9445a13f75c0a44c656803e6d3644b36c12e5956b968634b692447: Status 404 returned error can't find the container with id b0b106acdd9445a13f75c0a44c656803e6d3644b36c12e5956b968634b692447
Oct 30 00:53:51.553245 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:51.553212    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:53:51.553339 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:51.553276    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:53:51.553339 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:51.553306    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:53:52.216473 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:52.216430    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq" event={"ID":"e6ae4638-c9e9-4d20-b699-feb3208662a2","Type":"ContainerStarted","Data":"44b2431d0824a1af95eab7cedd2fc0bd4c7690e383cf9c6338cb0e131aca4102"}
Oct 30 00:53:52.216840 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:52.216492    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq" event={"ID":"e6ae4638-c9e9-4d20-b699-feb3208662a2","Type":"ContainerStarted","Data":"b0b106acdd9445a13f75c0a44c656803e6d3644b36c12e5956b968634b692447"}
Oct 30 00:53:52.235893 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:52.235865    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:52.240702 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:52.240676    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:52.306361 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:52.306310    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq" podStartSLOduration=2.306295581 podStartE2EDuration="2.306295581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:52.294551195 +0000 UTC m=+744.632653104" watchObservedRunningTime="2026-10-30 00:53:52.306295581 +0000 UTC m=+744.644397538"
Oct 30 00:53:53.221310 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:53.221282    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:53.222286 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:53.222266    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:53:55.281066 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:55.281030    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz"]
Oct 30 00:53:55.285177 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:55.285151    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz"
Oct 30 00:53:55.302769 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:55.302744    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-72ed13-serving-cert\""
Oct 30 00:53:55.313357 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:55.313316    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz"]
Oct 30 00:53:55.394128 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:55.394097    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53ecb5dd-2b7f-4139-8463-06583186f2e0-maas-api-tls\") pod \"maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz\" (UID: \"53ecb5dd-2b7f-4139-8463-06583186f2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz"
Oct 30 00:53:55.394316 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:55.394146    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l8gh\" (UniqueName: \"kubernetes.io/projected/53ecb5dd-2b7f-4139-8463-06583186f2e0-kube-api-access-6l8gh\") pod \"maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz\" (UID: \"53ecb5dd-2b7f-4139-8463-06583186f2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz"
Oct 30 00:53:55.394316 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:55.394261    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ecb5dd-2b7f-4139-8463-06583186f2e0-metrics-certs\") pod \"maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz\" (UID: \"53ecb5dd-2b7f-4139-8463-06583186f2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz"
Oct 30 00:53:55.394316 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:55.394290    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ecb5dd-2b7f-4139-8463-06583186f2e0-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz\" (UID: \"53ecb5dd-2b7f-4139-8463-06583186f2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz"
Oct 30 00:53:55.495586 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:55.495551    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ecb5dd-2b7f-4139-8463-06583186f2e0-metrics-certs\") pod \"maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz\" (UID: \"53ecb5dd-2b7f-4139-8463-06583186f2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz"
Oct 30 00:53:55.495784 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:55.495602    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ecb5dd-2b7f-4139-8463-06583186f2e0-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz\" (UID: \"53ecb5dd-2b7f-4139-8463-06583186f2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz"
Oct 30 00:53:55.495784 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:55.495716    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53ecb5dd-2b7f-4139-8463-06583186f2e0-maas-api-tls\") pod \"maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz\" (UID: \"53ecb5dd-2b7f-4139-8463-06583186f2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz"
Oct 30 00:53:55.495784 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:55.495753    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l8gh\" (UniqueName: \"kubernetes.io/projected/53ecb5dd-2b7f-4139-8463-06583186f2e0-kube-api-access-6l8gh\") pod \"maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz\" (UID: \"53ecb5dd-2b7f-4139-8463-06583186f2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz"
Oct 30 00:53:55.497193 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:55.497156    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ecb5dd-2b7f-4139-8463-06583186f2e0-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz\" (UID: \"53ecb5dd-2b7f-4139-8463-06583186f2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz"
Oct 30 00:53:55.498934 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:55.498872    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ecb5dd-2b7f-4139-8463-06583186f2e0-metrics-certs\") pod \"maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz\" (UID: \"53ecb5dd-2b7f-4139-8463-06583186f2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz"
Oct 30 00:53:55.499123 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:55.499099    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53ecb5dd-2b7f-4139-8463-06583186f2e0-maas-api-tls\") pod \"maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz\" (UID: \"53ecb5dd-2b7f-4139-8463-06583186f2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz"
Oct 30 00:53:55.518895 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:55.518859    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l8gh\" (UniqueName: \"kubernetes.io/projected/53ecb5dd-2b7f-4139-8463-06583186f2e0-kube-api-access-6l8gh\") pod \"maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz\" (UID: \"53ecb5dd-2b7f-4139-8463-06583186f2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz"
Oct 30 00:53:55.599201 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:55.599118    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz"
Oct 30 00:53:55.908836 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:55.908766    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz"]
Oct 30 00:53:55.912518 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:53:55.912486    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ecb5dd_2b7f_4139_8463_06583186f2e0.slice/crio-80b38e3420063ff73e7c00f7ebef2ec0eacc5574a8891745436b628642ab8368 WatchSource:0}: Error finding container 80b38e3420063ff73e7c00f7ebef2ec0eacc5574a8891745436b628642ab8368: Status 404 returned error can't find the container with id 80b38e3420063ff73e7c00f7ebef2ec0eacc5574a8891745436b628642ab8368
Oct 30 00:53:56.243823 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:56.243789    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz" event={"ID":"53ecb5dd-2b7f-4139-8463-06583186f2e0","Type":"ContainerStarted","Data":"80b38e3420063ff73e7c00f7ebef2ec0eacc5574a8891745436b628642ab8368"}
Oct 30 00:53:57.250836 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:57.250790    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz" event={"ID":"53ecb5dd-2b7f-4139-8463-06583186f2e0","Type":"ContainerStarted","Data":"e74ff560b30cb50706c437dcd5e171fab898e52fc4c2a288e043731c08318f35"}
Oct 30 00:53:57.251274 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:57.250965    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz"
Oct 30 00:53:57.293191 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:57.293140    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz" podStartSLOduration=1.952817938 podStartE2EDuration="2.29312671s" totalImagesPullingTime="340.308772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:57.291638696 +0000 UTC m=+749.629740606" watchObservedRunningTime="2026-10-30 00:53:57.29312671 +0000 UTC m=+749.631228619"
Oct 30 00:53:58.826467 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:58.826434    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69"]
Oct 30 00:53:58.830212 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:58.830187    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69"
Oct 30 00:53:58.850809 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:58.850765    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69"]
Oct 30 00:53:58.893945 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:58.893917    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-39b241\""
Oct 30 00:53:58.894126 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:58.893988    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-39b241-dockercfg-zdw98\""
Oct 30 00:53:58.930235 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:58.930189    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99nxn\" (UniqueName: \"kubernetes.io/projected/9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b-kube-api-access-99nxn\") pod \"payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69\" (UID: \"9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69"
Oct 30 00:53:58.930381 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:58.930321    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69\" (UID: \"9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69"
Oct 30 00:53:59.031541 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:59.031508    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69\" (UID: \"9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69"
Oct 30 00:53:59.031727 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:59.031631    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99nxn\" (UniqueName: \"kubernetes.io/projected/9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b-kube-api-access-99nxn\") pod \"payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69\" (UID: \"9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69"
Oct 30 00:53:59.032331 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:59.032308    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69\" (UID: \"9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69"
Oct 30 00:53:59.053489 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:59.053459    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99nxn\" (UniqueName: \"kubernetes.io/projected/9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b-kube-api-access-99nxn\") pod \"payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69\" (UID: \"9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69"
Oct 30 00:53:59.145136 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:59.145023    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69"
Oct 30 00:53:59.392310 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:59.392276    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz"]
Oct 30 00:53:59.397272 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:59.397218    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz"
Oct 30 00:53:59.411395 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:59.411371    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-664c2b-serving-cert\""
Oct 30 00:53:59.420191 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:59.420163    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz"]
Oct 30 00:53:59.434205 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:59.434153    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b52d59-2121-4a46-b9a4-8c80742be6c3-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz\" (UID: \"19b52d59-2121-4a46-b9a4-8c80742be6c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz"
Oct 30 00:53:59.434383 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:59.434282    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19b52d59-2121-4a46-b9a4-8c80742be6c3-metrics-certs\") pod \"maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz\" (UID: \"19b52d59-2121-4a46-b9a4-8c80742be6c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz"
Oct 30 00:53:59.434383 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:59.434316    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9gqg\" (UniqueName: \"kubernetes.io/projected/19b52d59-2121-4a46-b9a4-8c80742be6c3-kube-api-access-t9gqg\") pod \"maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz\" (UID: \"19b52d59-2121-4a46-b9a4-8c80742be6c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz"
Oct 30 00:53:59.434503 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:59.434437    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19b52d59-2121-4a46-b9a4-8c80742be6c3-maas-api-tls\") pod \"maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz\" (UID: \"19b52d59-2121-4a46-b9a4-8c80742be6c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz"
Oct 30 00:53:59.457552 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:59.457520    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69"]
Oct 30 00:53:59.462313 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:53:59.462268    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dbd469d_d1f9_4a08_bdd8_6f4a29848e6b.slice/crio-dc4cce0f24ce705df339a207c2d386821931a795bca74527d7b083b54f89ab6d WatchSource:0}: Error finding container dc4cce0f24ce705df339a207c2d386821931a795bca74527d7b083b54f89ab6d: Status 404 returned error can't find the container with id dc4cce0f24ce705df339a207c2d386821931a795bca74527d7b083b54f89ab6d
Oct 30 00:53:59.535852 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:59.535828    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b52d59-2121-4a46-b9a4-8c80742be6c3-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz\" (UID: \"19b52d59-2121-4a46-b9a4-8c80742be6c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz"
Oct 30 00:53:59.535945 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:59.535875    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19b52d59-2121-4a46-b9a4-8c80742be6c3-metrics-certs\") pod \"maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz\" (UID: \"19b52d59-2121-4a46-b9a4-8c80742be6c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz"
Oct 30 00:53:59.535945 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:59.535897    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9gqg\" (UniqueName: \"kubernetes.io/projected/19b52d59-2121-4a46-b9a4-8c80742be6c3-kube-api-access-t9gqg\") pod \"maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz\" (UID: \"19b52d59-2121-4a46-b9a4-8c80742be6c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz"
Oct 30 00:53:59.536013 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:59.535973    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19b52d59-2121-4a46-b9a4-8c80742be6c3-maas-api-tls\") pod \"maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz\" (UID: \"19b52d59-2121-4a46-b9a4-8c80742be6c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz"
Oct 30 00:53:59.536557 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:59.536534    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b52d59-2121-4a46-b9a4-8c80742be6c3-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz\" (UID: \"19b52d59-2121-4a46-b9a4-8c80742be6c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz"
Oct 30 00:53:59.538350 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:59.538330    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19b52d59-2121-4a46-b9a4-8c80742be6c3-metrics-certs\") pod \"maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz\" (UID: \"19b52d59-2121-4a46-b9a4-8c80742be6c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz"
Oct 30 00:53:59.538350 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:59.538338    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19b52d59-2121-4a46-b9a4-8c80742be6c3-maas-api-tls\") pod \"maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz\" (UID: \"19b52d59-2121-4a46-b9a4-8c80742be6c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz"
Oct 30 00:53:59.560737 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:59.560699    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9gqg\" (UniqueName: \"kubernetes.io/projected/19b52d59-2121-4a46-b9a4-8c80742be6c3-kube-api-access-t9gqg\") pod \"maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz\" (UID: \"19b52d59-2121-4a46-b9a4-8c80742be6c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz"
Oct 30 00:53:59.711215 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:53:59.711176    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz"
Oct 30 00:54:00.009905 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:00.009816    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz"]
Oct 30 00:54:00.014700 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:54:00.014672    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b52d59_2121_4a46_b9a4_8c80742be6c3.slice/crio-78244041ced506bcdf4b16573e3c078dc565c9a90ad6ed2583cf195d86ec1af0 WatchSource:0}: Error finding container 78244041ced506bcdf4b16573e3c078dc565c9a90ad6ed2583cf195d86ec1af0: Status 404 returned error can't find the container with id 78244041ced506bcdf4b16573e3c078dc565c9a90ad6ed2583cf195d86ec1af0
Oct 30 00:54:00.268414 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:00.268328    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz" event={"ID":"19b52d59-2121-4a46-b9a4-8c80742be6c3","Type":"ContainerStarted","Data":"78244041ced506bcdf4b16573e3c078dc565c9a90ad6ed2583cf195d86ec1af0"}
Oct 30 00:54:00.269959 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:00.269932    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69" event={"ID":"9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b","Type":"ContainerStarted","Data":"8916621cb9bdd2444ab199e5f6ae7931aa8422cd534e82929bc6bd8a9ba7f794"}
Oct 30 00:54:00.270128 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:00.269979    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69" event={"ID":"9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b","Type":"ContainerStarted","Data":"dc4cce0f24ce705df339a207c2d386821931a795bca74527d7b083b54f89ab6d"}
Oct 30 00:54:00.270128 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:00.270033    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69"
Oct 30 00:54:00.307981 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:00.307803    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69" podStartSLOduration=2.307786763 podStartE2EDuration="2.307786763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:00.306650845 +0000 UTC m=+752.644752756" watchObservedRunningTime="2026-10-30 00:54:00.307786763 +0000 UTC m=+752.645888670"
Oct 30 00:54:01.276225 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:01.276188    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz" event={"ID":"19b52d59-2121-4a46-b9a4-8c80742be6c3","Type":"ContainerStarted","Data":"313754e899fd358e7b5f6b067075a4bfec89221156ca30ac0dec298d2e13a248"}
Oct 30 00:54:01.320758 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:01.320695    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz" podStartSLOduration=1.929728437 podStartE2EDuration="2.320675151s" totalImagesPullingTime="390.946714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:01.319304025 +0000 UTC m=+753.657405936" watchObservedRunningTime="2026-10-30 00:54:01.320675151 +0000 UTC m=+753.658777063"
Oct 30 00:54:02.280858 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:02.280822    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz"
Oct 30 00:54:03.260975 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:03.260944    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz"
Oct 30 00:54:03.288910 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:03.288879    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp"]
Oct 30 00:54:03.292979 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:03.292962    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp"
Oct 30 00:54:03.309670 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:03.309642    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-39b241-serving-cert\""
Oct 30 00:54:03.315266 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:03.315237    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp"]
Oct 30 00:54:03.375062 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:03.375023    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-metrics-certs\") pod \"maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp\" (UID: \"5bf2af8d-0d32-4191-ac7a-69f8aef47e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp"
Oct 30 00:54:03.375261 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:03.375060    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl7n6\" (UniqueName: \"kubernetes.io/projected/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-kube-api-access-xl7n6\") pod \"maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp\" (UID: \"5bf2af8d-0d32-4191-ac7a-69f8aef47e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp"
Oct 30 00:54:03.375326 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:03.375277    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp\" (UID: \"5bf2af8d-0d32-4191-ac7a-69f8aef47e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp"
Oct 30 00:54:03.375521 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:03.375495    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp\" (UID: \"5bf2af8d-0d32-4191-ac7a-69f8aef47e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp"
Oct 30 00:54:03.477232 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:03.477199    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp\" (UID: \"5bf2af8d-0d32-4191-ac7a-69f8aef47e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp"
Oct 30 00:54:03.477414 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:03.477300    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-metrics-certs\") pod \"maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp\" (UID: \"5bf2af8d-0d32-4191-ac7a-69f8aef47e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp"
Oct 30 00:54:03.477414 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:03.477323    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl7n6\" (UniqueName: \"kubernetes.io/projected/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-kube-api-access-xl7n6\") pod \"maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp\" (UID: \"5bf2af8d-0d32-4191-ac7a-69f8aef47e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp"
Oct 30 00:54:03.477414 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:03.477354    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp\" (UID: \"5bf2af8d-0d32-4191-ac7a-69f8aef47e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp"
Oct 30 00:54:03.478097 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:03.478057    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp\" (UID: \"5bf2af8d-0d32-4191-ac7a-69f8aef47e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp"
Oct 30 00:54:03.479875 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:03.479847    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp\" (UID: \"5bf2af8d-0d32-4191-ac7a-69f8aef47e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp"
Oct 30 00:54:03.479982 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:03.479847    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-metrics-certs\") pod \"maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp\" (UID: \"5bf2af8d-0d32-4191-ac7a-69f8aef47e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp"
Oct 30 00:54:03.506363 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:03.506329    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl7n6\" (UniqueName: \"kubernetes.io/projected/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-kube-api-access-xl7n6\") pod \"maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp\" (UID: \"5bf2af8d-0d32-4191-ac7a-69f8aef47e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp"
Oct 30 00:54:03.606589 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:03.606502    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp"
Oct 30 00:54:03.906724 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:03.906633    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp"]
Oct 30 00:54:03.910880 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:54:03.910849    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf2af8d_0d32_4191_ac7a_69f8aef47e1a.slice/crio-1c23455cd339eff0c2b023a2965e7b4479423e70007983aa3904d62ccdb37f02 WatchSource:0}: Error finding container 1c23455cd339eff0c2b023a2965e7b4479423e70007983aa3904d62ccdb37f02: Status 404 returned error can't find the container with id 1c23455cd339eff0c2b023a2965e7b4479423e70007983aa3904d62ccdb37f02
Oct 30 00:54:04.295228 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:04.295136    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp" event={"ID":"5bf2af8d-0d32-4191-ac7a-69f8aef47e1a","Type":"ContainerStarted","Data":"1c23455cd339eff0c2b023a2965e7b4479423e70007983aa3904d62ccdb37f02"}
Oct 30 00:54:05.301600 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:05.301560    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp" event={"ID":"5bf2af8d-0d32-4191-ac7a-69f8aef47e1a","Type":"ContainerStarted","Data":"83cd20bdbfa59679eb170fdb679c0e888316266bb8210835fcf01012916ce5c1"}
Oct 30 00:54:05.302014 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:05.301650    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp"
Oct 30 00:54:05.345608 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:05.345554    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp" podStartSLOduration=2.037703743 podStartE2EDuration="2.345541062s" totalImagesPullingTime="307.837319ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:05.342910962 +0000 UTC m=+757.681012873" watchObservedRunningTime="2026-10-30 00:54:05.345541062 +0000 UTC m=+757.683642972"
Oct 30 00:54:06.803394 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:06.803355    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"]
Oct 30 00:54:06.803831 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:06.803564    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b" podUID="96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93" containerName="maas-api" containerID="cri-o://63f40ed8ca8587bbc39b58189917103eab1c51cac39f1f1d5efcb96fbda45e33" gracePeriod=30
Oct 30 00:54:07.113797 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:07.113770    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"
Oct 30 00:54:07.216353 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:07.216320    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-openshift-service-ca\") pod \"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93\" (UID: \"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93\") "
Oct 30 00:54:07.216497 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:07.216374    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-maas-api-tls\") pod \"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93\" (UID: \"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93\") "
Oct 30 00:54:07.216575 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:07.216557    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-kube-api-access-7wcjc\" (UniqueName: \"kubernetes.io/projected/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-kube-api-access-7wcjc\") pod \"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93\" (UID: \"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93\") "
Oct 30 00:54:07.216625 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:07.216605    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-metrics-certs\" (UniqueName: \"kubernetes.io/secret/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-metrics-certs\") pod \"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93\" (UID: \"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93\") "
Oct 30 00:54:07.216760 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:07.216733    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-openshift-service-ca" pod "96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93" (UID: "96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:54:07.216924 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:07.216909    4864 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-openshift-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:54:07.218591 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:07.218561    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-kube-api-access-7wcjc" pod "96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93" (UID: "96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93"). InnerVolumeSpecName "kube-api-access-7wcjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:54:07.218698 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:07.218583    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-maas-api-tls" pod "96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93" (UID: "96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:54:07.218698 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:07.218685    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-metrics-certs" pod "96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93" (UID: "96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:54:07.314536 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:07.314499    4864 generic.go:356] "Generic (PLEG): container finished" podID="96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93" containerID="63f40ed8ca8587bbc39b58189917103eab1c51cac39f1f1d5efcb96fbda45e33" exitCode=0
Oct 30 00:54:07.314706 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:07.314560    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b" event={"ID":"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93","Type":"ContainerDied","Data":"63f40ed8ca8587bbc39b58189917103eab1c51cac39f1f1d5efcb96fbda45e33"}
Oct 30 00:54:07.314706 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:07.314576    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"
Oct 30 00:54:07.314706 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:07.314692    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b" event={"ID":"96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93","Type":"ContainerDied","Data":"4a69d6ded2e4bdd3afc5cde3b734f11c1fe412596377d66e7a5e1ca63128c496"}
Oct 30 00:54:07.314881 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:07.314725    4864 scope.go:122] "RemoveContainer" containerID="63f40ed8ca8587bbc39b58189917103eab1c51cac39f1f1d5efcb96fbda45e33"
Oct 30 00:54:07.317701 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:07.317677    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wcjc\" (UniqueName: \"kubernetes.io/projected/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-kube-api-access-7wcjc\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:54:07.317701 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:07.317702    4864 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-metrics-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:54:07.317864 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:07.317712    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93-maas-api-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:54:07.326861 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:07.326846    4864 scope.go:122] "RemoveContainer" containerID="63f40ed8ca8587bbc39b58189917103eab1c51cac39f1f1d5efcb96fbda45e33"
Oct 30 00:54:07.327142 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:54:07.327115    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63f40ed8ca8587bbc39b58189917103eab1c51cac39f1f1d5efcb96fbda45e33\": container with ID starting with 63f40ed8ca8587bbc39b58189917103eab1c51cac39f1f1d5efcb96fbda45e33 not found: ID does not exist" containerID="63f40ed8ca8587bbc39b58189917103eab1c51cac39f1f1d5efcb96fbda45e33"
Oct 30 00:54:07.327516 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:07.327152    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63f40ed8ca8587bbc39b58189917103eab1c51cac39f1f1d5efcb96fbda45e33"} err="failed to get container status \"63f40ed8ca8587bbc39b58189917103eab1c51cac39f1f1d5efcb96fbda45e33\": rpc error: code = NotFound desc = could not find container \"63f40ed8ca8587bbc39b58189917103eab1c51cac39f1f1d5efcb96fbda45e33\": container with ID starting with 63f40ed8ca8587bbc39b58189917103eab1c51cac39f1f1d5efcb96fbda45e33 not found: ID does not exist"
Oct 30 00:54:07.383925 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:07.383836    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"]
Oct 30 00:54:07.386976 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:07.386949    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5e61cc8d-68cbf7b96d-mfp2b"]
Oct 30 00:54:07.792683 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:07.792652    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93" path="/var/lib/kubelet/pods/96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93/volumes"
Oct 30 00:54:08.292457 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:08.292427    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz"
Oct 30 00:54:09.964040 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:09.964004    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78"]
Oct 30 00:54:09.964649 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:09.964627    4864 cpu_manager.go:414] "containerMap: removing container" podUID="96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93" containerName="maas-api"
Oct 30 00:54:09.964736 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:09.964650    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93" containerName="maas-api"
Oct 30 00:54:09.964816 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:09.964802    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="96afb4dd-cfa3-4ce9-a74d-4c6ecfea0e93" containerName="maas-api"
Oct 30 00:54:09.969728 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:09.969707    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78"
Oct 30 00:54:10.002864 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:10.002834    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78"]
Oct 30 00:54:10.009326 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:10.009296    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-d5b2b8-serving-cert\""
Oct 30 00:54:10.044041 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:10.044006    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f110028-043c-470d-9724-290fcf719e99-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78\" (UID: \"3f110028-043c-470d-9724-290fcf719e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78"
Oct 30 00:54:10.044295 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:10.044188    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f110028-043c-470d-9724-290fcf719e99-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78\" (UID: \"3f110028-043c-470d-9724-290fcf719e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78"
Oct 30 00:54:10.044295 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:10.044236    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgtw5\" (UniqueName: \"kubernetes.io/projected/3f110028-043c-470d-9724-290fcf719e99-kube-api-access-qgtw5\") pod \"maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78\" (UID: \"3f110028-043c-470d-9724-290fcf719e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78"
Oct 30 00:54:10.044379 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:10.044303    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f110028-043c-470d-9724-290fcf719e99-metrics-certs\") pod \"maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78\" (UID: \"3f110028-043c-470d-9724-290fcf719e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78"
Oct 30 00:54:10.145346 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:10.145307    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f110028-043c-470d-9724-290fcf719e99-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78\" (UID: \"3f110028-043c-470d-9724-290fcf719e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78"
Oct 30 00:54:10.145538 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:10.145375    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgtw5\" (UniqueName: \"kubernetes.io/projected/3f110028-043c-470d-9724-290fcf719e99-kube-api-access-qgtw5\") pod \"maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78\" (UID: \"3f110028-043c-470d-9724-290fcf719e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78"
Oct 30 00:54:10.145538 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:10.145440    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f110028-043c-470d-9724-290fcf719e99-metrics-certs\") pod \"maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78\" (UID: \"3f110028-043c-470d-9724-290fcf719e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78"
Oct 30 00:54:10.145538 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:10.145533    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f110028-043c-470d-9724-290fcf719e99-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78\" (UID: \"3f110028-043c-470d-9724-290fcf719e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78"
Oct 30 00:54:10.146338 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:10.146307    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f110028-043c-470d-9724-290fcf719e99-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78\" (UID: \"3f110028-043c-470d-9724-290fcf719e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78"
Oct 30 00:54:10.148015 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:10.147985    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f110028-043c-470d-9724-290fcf719e99-metrics-certs\") pod \"maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78\" (UID: \"3f110028-043c-470d-9724-290fcf719e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78"
Oct 30 00:54:10.148137 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:10.148028    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f110028-043c-470d-9724-290fcf719e99-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78\" (UID: \"3f110028-043c-470d-9724-290fcf719e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78"
Oct 30 00:54:10.182764 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:10.182727    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgtw5\" (UniqueName: \"kubernetes.io/projected/3f110028-043c-470d-9724-290fcf719e99-kube-api-access-qgtw5\") pod \"maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78\" (UID: \"3f110028-043c-470d-9724-290fcf719e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78"
Oct 30 00:54:10.283865 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:10.283783    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78"
Oct 30 00:54:10.597516 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:10.597435    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78"]
Oct 30 00:54:10.605780 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:54:10.605741    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f110028_043c_470d_9724_290fcf719e99.slice/crio-8b9953247357c4d29774d2f31f382ce5eab657256cb2814f0e8cce43662a31af WatchSource:0}: Error finding container 8b9953247357c4d29774d2f31f382ce5eab657256cb2814f0e8cce43662a31af: Status 404 returned error can't find the container with id 8b9953247357c4d29774d2f31f382ce5eab657256cb2814f0e8cce43662a31af
Oct 30 00:54:11.283300 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:11.283265    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69"
Oct 30 00:54:11.312336 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:11.312306    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp"
Oct 30 00:54:11.343770 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:11.343710    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78" event={"ID":"3f110028-043c-470d-9724-290fcf719e99","Type":"ContainerStarted","Data":"7dfea5dfb0f6d65f86a71b103422019e549fac4ac1f64c7ac3502e427cf35a22"}
Oct 30 00:54:11.343947 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:11.343801    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78" event={"ID":"3f110028-043c-470d-9724-290fcf719e99","Type":"ContainerStarted","Data":"8b9953247357c4d29774d2f31f382ce5eab657256cb2814f0e8cce43662a31af"}
Oct 30 00:54:11.344093 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:11.344051    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78"
Oct 30 00:54:11.439977 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:11.439913    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78" podStartSLOduration=2.160799622 podStartE2EDuration="2.439897376s" totalImagesPullingTime="279.097754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:11.436026354 +0000 UTC m=+763.774128289" watchObservedRunningTime="2026-10-30 00:54:11.439897376 +0000 UTC m=+763.777999287"
Oct 30 00:54:13.196242 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:13.196209    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:54:13.202578 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:13.196277    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c" podUID="c08db946-5b81-477f-bc5a-8537b0385100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:54:13.368670 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:13.356423    4864 generic.go:356] "Generic (PLEG): container finished" podID="c08db946-5b81-477f-bc5a-8537b0385100" containerID="026f8125a099382faf1e5739df3b1d06eb4ecec768b5049395b8d1f2edc8fd4a" exitCode=0
Oct 30 00:54:13.368670 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:13.356535    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c" event={"ID":"c08db946-5b81-477f-bc5a-8537b0385100","Type":"ContainerDied","Data":"026f8125a099382faf1e5739df3b1d06eb4ecec768b5049395b8d1f2edc8fd4a"}
Oct 30 00:54:13.368670 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:13.356969    4864 scope.go:122] "RemoveContainer" containerID="026f8125a099382faf1e5739df3b1d06eb4ecec768b5049395b8d1f2edc8fd4a"
Oct 30 00:54:13.381749 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:13.381713    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:54:13.381889 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:13.381805    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:54:13.381889 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:13.381852    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:54:14.400060 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:14.400024    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c" event={"ID":"c08db946-5b81-477f-bc5a-8537b0385100","Type":"ContainerStarted","Data":"5a1f545e210433161c25127fa527a89e75af7995fc252f86da6e99a61ed3436d"}
Oct 30 00:54:15.352311 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:15.352276    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:54:15.352470 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:15.352427    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 00:54:15.352532 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:15.352466    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c" podUID="c08db946-5b81-477f-bc5a-8537b0385100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 00:54:16.352242 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:16.352208    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 00:54:16.352608 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:16.352263    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c" podUID="c08db946-5b81-477f-bc5a-8537b0385100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 00:54:17.351728 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:17.351688    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 00:54:17.352035 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:17.351992    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c" podUID="c08db946-5b81-477f-bc5a-8537b0385100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 00:54:17.355407 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:17.355384    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78"
Oct 30 00:54:18.294096 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:18.291721    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"]
Oct 30 00:54:18.294580 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:18.294546    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w" podUID="0225b6be-e7fc-4b20-a524-9ce6e6135c1f" containerName="istio-proxy" containerID="cri-o://41cbecbc2d99a2ec292415718c442171ae8be9a16835b54fe9194d5307454a7e" gracePeriod=30
Oct 30 00:54:18.352466 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:18.352429    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 00:54:18.352627 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:18.352480    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c" podUID="c08db946-5b81-477f-bc5a-8537b0385100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 00:54:19.041645 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:54:19.041608    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:54:19.279762 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:19.278355    4864 patch_prober.go:29] interesting pod/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 00:54:19.279762 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:19.278431    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w" podUID="0225b6be-e7fc-4b20-a524-9ce6e6135c1f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 00:54:19.352377 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:19.352303    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 00:54:19.352527 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:19.352380    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c" podUID="c08db946-5b81-477f-bc5a-8537b0385100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 00:54:20.355971 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:20.355888    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:54:20.356400 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:20.356280    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:54:20.356919 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:20.356903    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 00:54:20.439507 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:20.439475    4864 generic.go:356] "Generic (PLEG): container finished" podID="e6ae4638-c9e9-4d20-b699-feb3208662a2" containerID="44b2431d0824a1af95eab7cedd2fc0bd4c7690e383cf9c6338cb0e131aca4102" exitCode=0
Oct 30 00:54:20.439664 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:20.439545    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq" event={"ID":"e6ae4638-c9e9-4d20-b699-feb3208662a2","Type":"ContainerDied","Data":"44b2431d0824a1af95eab7cedd2fc0bd4c7690e383cf9c6338cb0e131aca4102"}
Oct 30 00:54:20.440100 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:20.440083    4864 scope.go:122] "RemoveContainer" containerID="44b2431d0824a1af95eab7cedd2fc0bd4c7690e383cf9c6338cb0e131aca4102"
Oct 30 00:54:20.441252 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:20.441227    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:54:20.441310 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:20.441288    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:54:20.441345 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:20.441320    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:54:21.446504 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:21.446412    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq" event={"ID":"e6ae4638-c9e9-4d20-b699-feb3208662a2","Type":"ContainerStarted","Data":"b0aa74b32daead2adef84a913717fb332f4c7daf40684bcefd534bbb06aa77af"}
Oct 30 00:54:22.235160 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:22.235119    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:54:22.236367 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:22.236340    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 00:54:22.236482 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:22.236401    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq" podUID="e6ae4638-c9e9-4d20-b699-feb3208662a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 00:54:23.221371 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.221339    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:54:23.235974 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.235939    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 00:54:23.236191 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.235999    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq" podUID="e6ae4638-c9e9-4d20-b699-feb3208662a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 00:54:23.483169 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.476113    4864 generic.go:356] "Generic (PLEG): container finished" podID="0225b6be-e7fc-4b20-a524-9ce6e6135c1f" containerID="41cbecbc2d99a2ec292415718c442171ae8be9a16835b54fe9194d5307454a7e" exitCode=0
Oct 30 00:54:23.483169 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.477675    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w" event={"ID":"0225b6be-e7fc-4b20-a524-9ce6e6135c1f","Type":"ContainerDied","Data":"41cbecbc2d99a2ec292415718c442171ae8be9a16835b54fe9194d5307454a7e"}
Oct 30 00:54:23.483169 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.477754    4864 scope.go:122] "RemoveContainer" containerID="74e0ae4b73aca38da9accc308916f718d7cd9cbc579d2a6a1ca8ed14db283510"
Oct 30 00:54:23.622917 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.622886    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:54:23.635153 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.635123    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-credential-socket\") pod \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") "
Oct 30 00:54:23.635340 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.635190    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-data\") pod \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") "
Oct 30 00:54:23.635340 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.635223    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-kube-api-access-jtlpd\" (UniqueName: \"kubernetes.io/projected/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-kube-api-access-jtlpd\") pod \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") "
Oct 30 00:54:23.635340 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.635248    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-envoy\") pod \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") "
Oct 30 00:54:23.635340 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.635273    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istiod-ca-cert\") pod \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") "
Oct 30 00:54:23.635340 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.635317    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-podinfo\") pod \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") "
Oct 30 00:54:23.635555 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.635348    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-workload-socket\") pod \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") "
Oct 30 00:54:23.635555 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.635367    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-workload-certs\") pod \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") "
Oct 30 00:54:23.635555 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.635405    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-token\" (UniqueName: \"kubernetes.io/projected/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-token\") pod \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\" (UID: \"0225b6be-e7fc-4b20-a524-9ce6e6135c1f\") "
Oct 30 00:54:23.636402 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.636337    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-data" pod "0225b6be-e7fc-4b20-a524-9ce6e6135c1f" (UID: "0225b6be-e7fc-4b20-a524-9ce6e6135c1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:54:23.636626 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.636599    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-credential-socket" pod "0225b6be-e7fc-4b20-a524-9ce6e6135c1f" (UID: "0225b6be-e7fc-4b20-a524-9ce6e6135c1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:54:23.636626 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.636604    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-workload-socket" pod "0225b6be-e7fc-4b20-a524-9ce6e6135c1f" (UID: "0225b6be-e7fc-4b20-a524-9ce6e6135c1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:54:23.636967 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.636825    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-workload-certs" pod "0225b6be-e7fc-4b20-a524-9ce6e6135c1f" (UID: "0225b6be-e7fc-4b20-a524-9ce6e6135c1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:54:23.636967 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.636943    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istiod-ca-cert" pod "0225b6be-e7fc-4b20-a524-9ce6e6135c1f" (UID: "0225b6be-e7fc-4b20-a524-9ce6e6135c1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:54:23.638063 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.638034    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-envoy" pod "0225b6be-e7fc-4b20-a524-9ce6e6135c1f" (UID: "0225b6be-e7fc-4b20-a524-9ce6e6135c1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:54:23.638299 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.638270    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-token" pod "0225b6be-e7fc-4b20-a524-9ce6e6135c1f" (UID: "0225b6be-e7fc-4b20-a524-9ce6e6135c1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:54:23.638398 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.638301    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-podinfo" pod "0225b6be-e7fc-4b20-a524-9ce6e6135c1f" (UID: "0225b6be-e7fc-4b20-a524-9ce6e6135c1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:54:23.638608 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.638587    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-kube-api-access-jtlpd" pod "0225b6be-e7fc-4b20-a524-9ce6e6135c1f" (UID: "0225b6be-e7fc-4b20-a524-9ce6e6135c1f"). InnerVolumeSpecName "kube-api-access-jtlpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:54:23.736581 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.736492    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:54:23.736581 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.736525    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:54:23.736581 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.736537    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:54:23.736581 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.736547    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtlpd\" (UniqueName: \"kubernetes.io/projected/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-kube-api-access-jtlpd\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:54:23.736581 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.736558    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:54:23.736581 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.736567    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:54:23.736581 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.736574    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:54:23.736581 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.736582    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:54:23.736948 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:23.736590    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0225b6be-e7fc-4b20-a524-9ce6e6135c1f-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:54:24.236373 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:24.236337    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 00:54:24.236775 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:24.236397    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq" podUID="e6ae4638-c9e9-4d20-b699-feb3208662a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 00:54:24.487468 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:24.487382    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"
Oct 30 00:54:24.487616 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:24.487372    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w" event={"ID":"0225b6be-e7fc-4b20-a524-9ce6e6135c1f","Type":"ContainerDied","Data":"5014f8ad9c496de8a08c398fdfaafc4e0534e97f2238b194d7dabe613507d8c2"}
Oct 30 00:54:24.487616 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:24.487562    4864 scope.go:122] "RemoveContainer" containerID="41cbecbc2d99a2ec292415718c442171ae8be9a16835b54fe9194d5307454a7e"
Oct 30 00:54:24.586284 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:24.586242    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"]
Oct 30 00:54:24.589472 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:24.589433    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5e61cc8d-openshift-default-57d9449b6c-x2d8w"]
Oct 30 00:54:25.235620 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.235588    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 00:54:25.235801 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.235639    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq" podUID="e6ae4638-c9e9-4d20-b699-feb3208662a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 00:54:25.311131 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.311092    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq"]
Oct 30 00:54:25.311602 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.311586    4864 cpu_manager.go:414] "containerMap: removing container" podUID="0225b6be-e7fc-4b20-a524-9ce6e6135c1f" containerName="istio-proxy"
Oct 30 00:54:25.311644 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.311603    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0225b6be-e7fc-4b20-a524-9ce6e6135c1f" containerName="istio-proxy"
Oct 30 00:54:25.311644 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.311631    4864 cpu_manager.go:414] "containerMap: removing container" podUID="0225b6be-e7fc-4b20-a524-9ce6e6135c1f" containerName="istio-proxy"
Oct 30 00:54:25.311644 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.311636    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0225b6be-e7fc-4b20-a524-9ce6e6135c1f" containerName="istio-proxy"
Oct 30 00:54:25.311734 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.311696    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="0225b6be-e7fc-4b20-a524-9ce6e6135c1f" containerName="istio-proxy"
Oct 30 00:54:25.314802 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.314787    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq"
Oct 30 00:54:25.336716 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.336675    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq"]
Oct 30 00:54:25.346451 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.346420    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-9bfe36\""
Oct 30 00:54:25.352441 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.352413    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b674ae-7af3-4ef9-a04a-bea2b1071c1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq\" (UID: \"46b674ae-7af3-4ef9-a04a-bea2b1071c1d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq"
Oct 30 00:54:25.352572 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.352557    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txwnc\" (UniqueName: \"kubernetes.io/projected/46b674ae-7af3-4ef9-a04a-bea2b1071c1d-kube-api-access-txwnc\") pod \"payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq\" (UID: \"46b674ae-7af3-4ef9-a04a-bea2b1071c1d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq"
Oct 30 00:54:25.432658 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.432622    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx"]
Oct 30 00:54:25.433214 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.433199    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="0225b6be-e7fc-4b20-a524-9ce6e6135c1f" containerName="istio-proxy"
Oct 30 00:54:25.435443 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.435426    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx"
Oct 30 00:54:25.450803 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.450776    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-9bfe36-dockercfg-fmr6c\""
Oct 30 00:54:25.453583 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.453551    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dncx\" (UniqueName: \"kubernetes.io/projected/83de80bb-1c42-4450-9019-abc0e614280b-kube-api-access-4dncx\") pod \"payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx\" (UID: \"83de80bb-1c42-4450-9019-abc0e614280b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx"
Oct 30 00:54:25.454120 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.453631    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b674ae-7af3-4ef9-a04a-bea2b1071c1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq\" (UID: \"46b674ae-7af3-4ef9-a04a-bea2b1071c1d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq"
Oct 30 00:54:25.454120 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.453722    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83de80bb-1c42-4450-9019-abc0e614280b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx\" (UID: \"83de80bb-1c42-4450-9019-abc0e614280b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx"
Oct 30 00:54:25.454120 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.453771    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txwnc\" (UniqueName: \"kubernetes.io/projected/46b674ae-7af3-4ef9-a04a-bea2b1071c1d-kube-api-access-txwnc\") pod \"payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq\" (UID: \"46b674ae-7af3-4ef9-a04a-bea2b1071c1d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq"
Oct 30 00:54:25.454734 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.454712    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b674ae-7af3-4ef9-a04a-bea2b1071c1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq\" (UID: \"46b674ae-7af3-4ef9-a04a-bea2b1071c1d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq"
Oct 30 00:54:25.491180 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.481439    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx"]
Oct 30 00:54:25.510975 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.510945    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txwnc\" (UniqueName: \"kubernetes.io/projected/46b674ae-7af3-4ef9-a04a-bea2b1071c1d-kube-api-access-txwnc\") pod \"payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq\" (UID: \"46b674ae-7af3-4ef9-a04a-bea2b1071c1d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq"
Oct 30 00:54:25.554892 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.554855    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dncx\" (UniqueName: \"kubernetes.io/projected/83de80bb-1c42-4450-9019-abc0e614280b-kube-api-access-4dncx\") pod \"payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx\" (UID: \"83de80bb-1c42-4450-9019-abc0e614280b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx"
Oct 30 00:54:25.555056 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.554959    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83de80bb-1c42-4450-9019-abc0e614280b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx\" (UID: \"83de80bb-1c42-4450-9019-abc0e614280b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx"
Oct 30 00:54:25.555664 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.555644    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83de80bb-1c42-4450-9019-abc0e614280b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx\" (UID: \"83de80bb-1c42-4450-9019-abc0e614280b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx"
Oct 30 00:54:25.584108 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.584058    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dncx\" (UniqueName: \"kubernetes.io/projected/83de80bb-1c42-4450-9019-abc0e614280b-kube-api-access-4dncx\") pod \"payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx\" (UID: \"83de80bb-1c42-4450-9019-abc0e614280b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx"
Oct 30 00:54:25.628208 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.628174    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq"
Oct 30 00:54:25.748402 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.748188    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx"
Oct 30 00:54:25.796522 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.796487    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0225b6be-e7fc-4b20-a524-9ce6e6135c1f" path="/var/lib/kubelet/pods/0225b6be-e7fc-4b20-a524-9ce6e6135c1f/volumes"
Oct 30 00:54:25.983514 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:25.983480    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq"]
Oct 30 00:54:25.986171 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:54:25.986140    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b674ae_7af3_4ef9_a04a_bea2b1071c1d.slice/crio-5f4c40dfe3510a24ac859996623da1180333905c1950226daca36df9ce4b0964 WatchSource:0}: Error finding container 5f4c40dfe3510a24ac859996623da1180333905c1950226daca36df9ce4b0964: Status 404 returned error can't find the container with id 5f4c40dfe3510a24ac859996623da1180333905c1950226daca36df9ce4b0964
Oct 30 00:54:26.107828 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:26.107792    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx"]
Oct 30 00:54:26.115966 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:54:26.115919    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83de80bb_1c42_4450_9019_abc0e614280b.slice/crio-739ec9c8585d9b7f9523908b651a7bc38db058d537bf81aae0ef800d9f05d940 WatchSource:0}: Error finding container 739ec9c8585d9b7f9523908b651a7bc38db058d537bf81aae0ef800d9f05d940: Status 404 returned error can't find the container with id 739ec9c8585d9b7f9523908b651a7bc38db058d537bf81aae0ef800d9f05d940
Oct 30 00:54:26.235532 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:26.235498    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 00:54:26.235652 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:26.235543    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq" podUID="e6ae4638-c9e9-4d20-b699-feb3208662a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 00:54:26.520491 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:26.520450    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx" event={"ID":"83de80bb-1c42-4450-9019-abc0e614280b","Type":"ContainerStarted","Data":"253efa8f059d061db3bcd570c42249fd7beeb1ef8047262cf797fa5eee3163d8"}
Oct 30 00:54:26.520976 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:26.520520    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx" event={"ID":"83de80bb-1c42-4450-9019-abc0e614280b","Type":"ContainerStarted","Data":"739ec9c8585d9b7f9523908b651a7bc38db058d537bf81aae0ef800d9f05d940"}
Oct 30 00:54:26.520976 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:26.520669    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx"
Oct 30 00:54:26.522588 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:26.522563    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq" event={"ID":"46b674ae-7af3-4ef9-a04a-bea2b1071c1d","Type":"ContainerStarted","Data":"e991e8f9be961ecf8bb38224e4ed311bb6e308b0e082400b3cfe4f77593ed895"}
Oct 30 00:54:26.522715 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:26.522609    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq" event={"ID":"46b674ae-7af3-4ef9-a04a-bea2b1071c1d","Type":"ContainerStarted","Data":"5f4c40dfe3510a24ac859996623da1180333905c1950226daca36df9ce4b0964"}
Oct 30 00:54:26.522715 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:26.522678    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq"
Oct 30 00:54:26.559255 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:26.559203    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx" podStartSLOduration=1.559189583 podStartE2EDuration="1.559189583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:26.557372673 +0000 UTC m=+778.895474596" watchObservedRunningTime="2026-10-30 00:54:26.559189583 +0000 UTC m=+778.897291492"
Oct 30 00:54:26.603572 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:26.602762    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq" podStartSLOduration=1.6027442079999998 podStartE2EDuration="1.602744208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:26.601771362 +0000 UTC m=+778.939873400" watchObservedRunningTime="2026-10-30 00:54:26.602744208 +0000 UTC m=+778.940846119"
Oct 30 00:54:27.239826 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:27.239796    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:54:27.240809 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:27.240786    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 00:54:32.884996 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:32.884959    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s"]
Oct 30 00:54:32.889032 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:32.889013    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s"
Oct 30 00:54:32.902578 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:32.902558    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-9bfe36-serving-cert\""
Oct 30 00:54:32.915412 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:32.915378    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s"]
Oct 30 00:54:32.955490 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:32.955461    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-maas-api-tls\") pod \"maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s\" (UID: \"e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s"
Oct 30 00:54:32.955618 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:32.955527    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-metrics-certs\") pod \"maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s\" (UID: \"e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s"
Oct 30 00:54:32.955679 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:32.955628    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwvqh\" (UniqueName: \"kubernetes.io/projected/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-kube-api-access-wwvqh\") pod \"maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s\" (UID: \"e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s"
Oct 30 00:54:32.955716 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:32.955676    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s\" (UID: \"e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s"
Oct 30 00:54:33.057132 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:33.057067    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-metrics-certs\") pod \"maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s\" (UID: \"e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s"
Oct 30 00:54:33.057309 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:33.057180    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwvqh\" (UniqueName: \"kubernetes.io/projected/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-kube-api-access-wwvqh\") pod \"maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s\" (UID: \"e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s"
Oct 30 00:54:33.057309 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:33.057236    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s\" (UID: \"e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s"
Oct 30 00:54:33.057309 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:33.057281    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-maas-api-tls\") pod \"maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s\" (UID: \"e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s"
Oct 30 00:54:33.057985 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:33.057951    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s\" (UID: \"e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s"
Oct 30 00:54:33.059811 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:33.059785    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-metrics-certs\") pod \"maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s\" (UID: \"e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s"
Oct 30 00:54:33.060094 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:33.060060    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-maas-api-tls\") pod \"maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s\" (UID: \"e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s"
Oct 30 00:54:33.078655 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:33.078630    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwvqh\" (UniqueName: \"kubernetes.io/projected/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-kube-api-access-wwvqh\") pod \"maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s\" (UID: \"e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s"
Oct 30 00:54:33.201501 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:33.201465    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s"
Oct 30 00:54:33.503427 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:33.503341    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s"]
Oct 30 00:54:33.506708 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:54:33.506676    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ceca67_73d9_4d66_8fbf_c5af36ec1e7c.slice/crio-43b39e9e6e481a9198c46eafebd18fdef183469984518b693db53fc86cb2213c WatchSource:0}: Error finding container 43b39e9e6e481a9198c46eafebd18fdef183469984518b693db53fc86cb2213c: Status 404 returned error can't find the container with id 43b39e9e6e481a9198c46eafebd18fdef183469984518b693db53fc86cb2213c
Oct 30 00:54:33.560638 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:33.560603    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s" event={"ID":"e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c","Type":"ContainerStarted","Data":"43b39e9e6e481a9198c46eafebd18fdef183469984518b693db53fc86cb2213c"}
Oct 30 00:54:34.567496 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:34.567459    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s" event={"ID":"e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c","Type":"ContainerStarted","Data":"3d2cd75496e25b30666e8cbe951e60d725610a27657547993e87d2a3648ae4af"}
Oct 30 00:54:34.567912 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:34.567644    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s"
Oct 30 00:54:34.607260 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:34.607212    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s" podStartSLOduration=2.344761883 podStartE2EDuration="2.607196436s" totalImagesPullingTime="262.434553ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:34.605137789 +0000 UTC m=+786.943239712" watchObservedRunningTime="2026-10-30 00:54:34.607196436 +0000 UTC m=+786.945298405"
Oct 30 00:54:37.531040 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:37.531011    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx"
Oct 30 00:54:37.531446 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:37.531090    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq"
Oct 30 00:54:37.795446 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:37.795369    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"]
Oct 30 00:54:37.798589 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:37.798562    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:54:37.813252 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:37.813223    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d5b2b8\"/\"kube-root-ca.crt\""
Oct 30 00:54:37.813534 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:37.813291    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d5b2b8\"/\"openshift-service-ca.crt\""
Oct 30 00:54:37.813714 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:37.813519    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d5b2b8\"/\"simulator-w4-d5b2b8-kserve-self-signed-certs\""
Oct 30 00:54:37.813714 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:37.813563    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d5b2b8\"/\"default-dockercfg-f9x4g\""
Oct 30 00:54:37.817058 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:37.817027    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"]
Oct 30 00:54:37.907643 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:37.907600    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-home\") pod \"simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv\" (UID: \"3c545315-888c-4418-8348-b06a0080a599\") " pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:54:37.907643 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:37.907649    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqg7z\" (UniqueName: \"kubernetes.io/projected/3c545315-888c-4418-8348-b06a0080a599-kube-api-access-mqg7z\") pod \"simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv\" (UID: \"3c545315-888c-4418-8348-b06a0080a599\") " pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:54:37.907863 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:37.907750    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-model-cache\") pod \"simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv\" (UID: \"3c545315-888c-4418-8348-b06a0080a599\") " pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:54:37.907863 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:37.907785    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-dshm\") pod \"simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv\" (UID: \"3c545315-888c-4418-8348-b06a0080a599\") " pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:54:37.907935 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:37.907868    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-tmp-dir\") pod \"simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv\" (UID: \"3c545315-888c-4418-8348-b06a0080a599\") " pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:54:37.907935 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:37.907892    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c545315-888c-4418-8348-b06a0080a599-tls-certs\") pod \"simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv\" (UID: \"3c545315-888c-4418-8348-b06a0080a599\") " pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:54:38.008788 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:38.008752    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-model-cache\") pod \"simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv\" (UID: \"3c545315-888c-4418-8348-b06a0080a599\") " pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:54:38.008788 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:38.008791    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-dshm\") pod \"simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv\" (UID: \"3c545315-888c-4418-8348-b06a0080a599\") " pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:54:38.009031 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:38.008863    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-tmp-dir\") pod \"simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv\" (UID: \"3c545315-888c-4418-8348-b06a0080a599\") " pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:54:38.009031 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:38.008901    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c545315-888c-4418-8348-b06a0080a599-tls-certs\") pod \"simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv\" (UID: \"3c545315-888c-4418-8348-b06a0080a599\") " pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:54:38.009031 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:38.008978    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-home\") pod \"simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv\" (UID: \"3c545315-888c-4418-8348-b06a0080a599\") " pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:54:38.009031 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:38.009012    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqg7z\" (UniqueName: \"kubernetes.io/projected/3c545315-888c-4418-8348-b06a0080a599-kube-api-access-mqg7z\") pod \"simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv\" (UID: \"3c545315-888c-4418-8348-b06a0080a599\") " pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:54:38.009258 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:38.009242    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-model-cache\") pod \"simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv\" (UID: \"3c545315-888c-4418-8348-b06a0080a599\") " pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:54:38.009655 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:38.009295    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-tmp-dir\") pod \"simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv\" (UID: \"3c545315-888c-4418-8348-b06a0080a599\") " pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:54:38.009655 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:38.009398    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-home\") pod \"simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv\" (UID: \"3c545315-888c-4418-8348-b06a0080a599\") " pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:54:38.011253 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:38.011228    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-dshm\") pod \"simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv\" (UID: \"3c545315-888c-4418-8348-b06a0080a599\") " pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:54:38.011616 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:38.011598    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c545315-888c-4418-8348-b06a0080a599-tls-certs\") pod \"simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv\" (UID: \"3c545315-888c-4418-8348-b06a0080a599\") " pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:54:38.030386 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:38.030361    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqg7z\" (UniqueName: \"kubernetes.io/projected/3c545315-888c-4418-8348-b06a0080a599-kube-api-access-mqg7z\") pod \"simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv\" (UID: \"3c545315-888c-4418-8348-b06a0080a599\") " pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:54:38.112289 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:38.112197    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:54:38.412032 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:38.411952    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"]
Oct 30 00:54:38.416362 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:54:38.416332    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c545315_888c_4418_8348_b06a0080a599.slice/crio-2cc8784f938fb6217df39ffe422656e46f988f2778e92f6d276005bac9b3e7ce WatchSource:0}: Error finding container 2cc8784f938fb6217df39ffe422656e46f988f2778e92f6d276005bac9b3e7ce: Status 404 returned error can't find the container with id 2cc8784f938fb6217df39ffe422656e46f988f2778e92f6d276005bac9b3e7ce
Oct 30 00:54:38.590280 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:38.590238    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv" event={"ID":"3c545315-888c-4418-8348-b06a0080a599","Type":"ContainerStarted","Data":"b0d47757aff813bc96a7112adad5d899bb1544eccde69dad7d75065578426421"}
Oct 30 00:54:38.590685 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:38.590287    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv" event={"ID":"3c545315-888c-4418-8348-b06a0080a599","Type":"ContainerStarted","Data":"2cc8784f938fb6217df39ffe422656e46f988f2778e92f6d276005bac9b3e7ce"}
Oct 30 00:54:38.628772 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:38.628717    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv" podStartSLOduration=1.628702138 podStartE2EDuration="1.628702138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:38.628056221 +0000 UTC m=+790.966158130" watchObservedRunningTime="2026-10-30 00:54:38.628702138 +0000 UTC m=+790.966804048"
Oct 30 00:54:39.112718 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:39.112686    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:54:40.577523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:40.577498    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s"
Oct 30 00:54:48.112767 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:48.112731    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:54:48.125662 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:48.125639    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:54:48.653565 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:48.653533    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:54:49.039964 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:54:49.039929    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:54:57.750845 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.750811    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"]
Oct 30 00:54:57.754261 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.754240    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:54:57.768276 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.768249    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-39b241\"/\"default-dockercfg-nf2tx\""
Oct 30 00:54:57.778390 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.778356    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-39b241\"/\"kube-root-ca.crt\""
Oct 30 00:54:57.778535 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.778397    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-39b241\"/\"openshift-service-ca.crt\""
Oct 30 00:54:57.782584 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.782554    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"]
Oct 30 00:54:57.790066 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.790040    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-home\") pod \"premium-w3-39b241-kserve-54df87b4c8-5kjxj\" (UID: \"8b267884-baae-4ad3-a414-82467f7188b3\") " pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:54:57.790247 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.790099    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-tmp-dir\") pod \"premium-w3-39b241-kserve-54df87b4c8-5kjxj\" (UID: \"8b267884-baae-4ad3-a414-82467f7188b3\") " pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:54:57.790247 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.790157    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-dshm\") pod \"premium-w3-39b241-kserve-54df87b4c8-5kjxj\" (UID: \"8b267884-baae-4ad3-a414-82467f7188b3\") " pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:54:57.790247 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.790188    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8pxk\" (UniqueName: \"kubernetes.io/projected/8b267884-baae-4ad3-a414-82467f7188b3-kube-api-access-z8pxk\") pod \"premium-w3-39b241-kserve-54df87b4c8-5kjxj\" (UID: \"8b267884-baae-4ad3-a414-82467f7188b3\") " pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:54:57.790247 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.790227    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b267884-baae-4ad3-a414-82467f7188b3-tls-certs\") pod \"premium-w3-39b241-kserve-54df87b4c8-5kjxj\" (UID: \"8b267884-baae-4ad3-a414-82467f7188b3\") " pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:54:57.790482 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.790418    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-model-cache\") pod \"premium-w3-39b241-kserve-54df87b4c8-5kjxj\" (UID: \"8b267884-baae-4ad3-a414-82467f7188b3\") " pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:54:57.867876 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.867847    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-39b241\"/\"premium-w3-39b241-kserve-self-signed-certs\""
Oct 30 00:54:57.891261 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.891236    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-dshm\") pod \"premium-w3-39b241-kserve-54df87b4c8-5kjxj\" (UID: \"8b267884-baae-4ad3-a414-82467f7188b3\") " pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:54:57.891400 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.891279    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8pxk\" (UniqueName: \"kubernetes.io/projected/8b267884-baae-4ad3-a414-82467f7188b3-kube-api-access-z8pxk\") pod \"premium-w3-39b241-kserve-54df87b4c8-5kjxj\" (UID: \"8b267884-baae-4ad3-a414-82467f7188b3\") " pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:54:57.891400 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.891307    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b267884-baae-4ad3-a414-82467f7188b3-tls-certs\") pod \"premium-w3-39b241-kserve-54df87b4c8-5kjxj\" (UID: \"8b267884-baae-4ad3-a414-82467f7188b3\") " pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:54:57.891400 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.891350    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-model-cache\") pod \"premium-w3-39b241-kserve-54df87b4c8-5kjxj\" (UID: \"8b267884-baae-4ad3-a414-82467f7188b3\") " pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:54:57.891523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.891418    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-home\") pod \"premium-w3-39b241-kserve-54df87b4c8-5kjxj\" (UID: \"8b267884-baae-4ad3-a414-82467f7188b3\") " pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:54:57.891523 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.891440    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-tmp-dir\") pod \"premium-w3-39b241-kserve-54df87b4c8-5kjxj\" (UID: \"8b267884-baae-4ad3-a414-82467f7188b3\") " pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:54:57.891803 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.891775    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-tmp-dir\") pod \"premium-w3-39b241-kserve-54df87b4c8-5kjxj\" (UID: \"8b267884-baae-4ad3-a414-82467f7188b3\") " pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:54:57.891917 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.891815    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-model-cache\") pod \"premium-w3-39b241-kserve-54df87b4c8-5kjxj\" (UID: \"8b267884-baae-4ad3-a414-82467f7188b3\") " pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:54:57.891917 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.891842    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-home\") pod \"premium-w3-39b241-kserve-54df87b4c8-5kjxj\" (UID: \"8b267884-baae-4ad3-a414-82467f7188b3\") " pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:54:57.893567 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.893545    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-dshm\") pod \"premium-w3-39b241-kserve-54df87b4c8-5kjxj\" (UID: \"8b267884-baae-4ad3-a414-82467f7188b3\") " pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:54:57.893931 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.893915    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b267884-baae-4ad3-a414-82467f7188b3-tls-certs\") pod \"premium-w3-39b241-kserve-54df87b4c8-5kjxj\" (UID: \"8b267884-baae-4ad3-a414-82467f7188b3\") " pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:54:57.912500 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:57.912479    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8pxk\" (UniqueName: \"kubernetes.io/projected/8b267884-baae-4ad3-a414-82467f7188b3-kube-api-access-z8pxk\") pod \"premium-w3-39b241-kserve-54df87b4c8-5kjxj\" (UID: \"8b267884-baae-4ad3-a414-82467f7188b3\") " pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:54:58.082293 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:58.082196    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:54:58.405113 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:58.405016    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"]
Oct 30 00:54:58.408910 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:54:58.408881    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b267884_baae_4ad3_a414_82467f7188b3.slice/crio-d74099911399f014ffa618601db63083ad37bd8428ee134bedc5fa771061d727 WatchSource:0}: Error finding container d74099911399f014ffa618601db63083ad37bd8428ee134bedc5fa771061d727: Status 404 returned error can't find the container with id d74099911399f014ffa618601db63083ad37bd8428ee134bedc5fa771061d727
Oct 30 00:54:58.696834 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:58.696743    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj" event={"ID":"8b267884-baae-4ad3-a414-82467f7188b3","Type":"ContainerStarted","Data":"10ba0f967830bb399f6d3ea3615ba77b0b60b2e8c4fb47be3154669f50da7a02"}
Oct 30 00:54:58.696834 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:58.696790    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj" event={"ID":"8b267884-baae-4ad3-a414-82467f7188b3","Type":"ContainerStarted","Data":"d74099911399f014ffa618601db63083ad37bd8428ee134bedc5fa771061d727"}
Oct 30 00:54:58.734302 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:58.734246    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj" podStartSLOduration=1.734226653 podStartE2EDuration="1.734226653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:58.733915906 +0000 UTC m=+811.072017815" watchObservedRunningTime="2026-10-30 00:54:58.734226653 +0000 UTC m=+811.072328563"
Oct 30 00:54:59.083374 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:54:59.083334    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:55:02.719597 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.719561    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"]
Oct 30 00:55:02.723357 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.723334    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.736233 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.736210    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-72ed13-openshift-default-dockercfg-tjgpb\""
Oct 30 00:55:02.740939 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.740913    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-workload-certs\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.741055 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.741015    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-credential-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.741055 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.741041    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-data\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.741187 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.741119    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-token\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.741240 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.741221    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c965c38-598d-47ce-9104-5a0b85c6a925-istiod-ca-cert\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.741282 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.741268    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27hrx\" (UniqueName: \"kubernetes.io/projected/7c965c38-598d-47ce-9104-5a0b85c6a925-kube-api-access-27hrx\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.741378 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.741362    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-workload-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.741421 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.741408    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-envoy\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.741494 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.741477    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-podinfo\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.749734 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.749707    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"]
Oct 30 00:55:02.842501 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.842468    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-credential-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.842501 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.842504    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-data\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.842739 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.842521    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-token\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.842739 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.842553    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c965c38-598d-47ce-9104-5a0b85c6a925-istiod-ca-cert\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.842739 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.842593    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27hrx\" (UniqueName: \"kubernetes.io/projected/7c965c38-598d-47ce-9104-5a0b85c6a925-kube-api-access-27hrx\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.842739 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.842650    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-workload-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.842739 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.842703    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-envoy\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.842739 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.842738    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-podinfo\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.843027 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.842768    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-workload-certs\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.843027 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.842938    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-credential-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.843160 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.843137    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-data\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.843219 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.843173    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-workload-certs\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.843219 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.843204    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-workload-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.843445 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.843421    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c965c38-598d-47ce-9104-5a0b85c6a925-istiod-ca-cert\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.844969 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.844941    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-envoy\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.845214 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.845197    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-podinfo\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.863875 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.863844    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-token\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:02.863980 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:02.863908    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27hrx\" (UniqueName: \"kubernetes.io/projected/7c965c38-598d-47ce-9104-5a0b85c6a925-kube-api-access-27hrx\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:03.036176 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:03.036096    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:03.351996 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:03.351919    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"]
Oct 30 00:55:03.356110 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:55:03.356064    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c965c38_598d_47ce_9104_5a0b85c6a925.slice/crio-fae9f4ef2b1537c585318e22c37385b6c92a97016e55e7752175726d6889773e WatchSource:0}: Error finding container fae9f4ef2b1537c585318e22c37385b6c92a97016e55e7752175726d6889773e: Status 404 returned error can't find the container with id fae9f4ef2b1537c585318e22c37385b6c92a97016e55e7752175726d6889773e
Oct 30 00:55:03.358474 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:03.358447    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:55:03.358536 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:03.358509    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:55:03.358575 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:03.358536    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:55:03.728085 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:03.728037    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt" event={"ID":"7c965c38-598d-47ce-9104-5a0b85c6a925","Type":"ContainerStarted","Data":"169eef1a050b2df2b8fc45d99406a76467632798d355bdf7db38a6e7acb30a5e"}
Oct 30 00:55:03.728574 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:03.728126    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt" event={"ID":"7c965c38-598d-47ce-9104-5a0b85c6a925","Type":"ContainerStarted","Data":"fae9f4ef2b1537c585318e22c37385b6c92a97016e55e7752175726d6889773e"}
Oct 30 00:55:03.797149 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:03.797104    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt" podStartSLOduration=1.7970894560000001 podStartE2EDuration="1.797089456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:03.794741461 +0000 UTC m=+816.132843375" watchObservedRunningTime="2026-10-30 00:55:03.797089456 +0000 UTC m=+816.135191357"
Oct 30 00:55:04.036341 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:04.036251    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:04.037849 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:04.037826    4864 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 00:55:04.037947 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:04.037883    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt" podUID="7c965c38-598d-47ce-9104-5a0b85c6a925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 00:55:05.036445 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:05.036412    4864 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 00:55:05.036824 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:05.036463    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt" podUID="7c965c38-598d-47ce-9104-5a0b85c6a925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 00:55:06.036646 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:06.036609    4864 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 00:55:06.037031 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:06.036664    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt" podUID="7c965c38-598d-47ce-9104-5a0b85c6a925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 00:55:07.037330 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:07.037300    4864 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 00:55:07.037731 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:07.037352    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt" podUID="7c965c38-598d-47ce-9104-5a0b85c6a925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 00:55:08.037008 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:08.036973    4864 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 00:55:08.037221 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:08.037047    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt" podUID="7c965c38-598d-47ce-9104-5a0b85c6a925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 00:55:08.083932 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:08.083138    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:55:08.100422 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:08.100391    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:55:08.793667 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:08.793633    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:55:09.037347 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:09.037315    4864 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 00:55:09.037553 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:09.037380    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt" podUID="7c965c38-598d-47ce-9104-5a0b85c6a925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 00:55:10.071310 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:10.071280    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:10.071837 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:10.071714    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:10.072717 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:10.072695    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 00:55:12.213127 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.213093    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"]
Oct 30 00:55:12.218394 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.218372    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.231335 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.231308    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d5b2b8-openshift-default-dockercfg-br5n4\""
Oct 30 00:55:12.244216 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.244189    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"]
Oct 30 00:55:12.259956 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.259924    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-token\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.260115 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.259977    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-podinfo\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.260115 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.260062    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-workload-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.260330 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.260313    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-envoy\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.260474 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.260447    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-workload-certs\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.260515 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.260499    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-credential-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.260639 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.260620    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-data\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.260705 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.260664    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istiod-ca-cert\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.260752 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.260721    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4tmv\" (UniqueName: \"kubernetes.io/projected/b460b90d-0d55-44af-a4b9-2e70b3a7b679-kube-api-access-f4tmv\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.362270 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.362222    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-token\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.362458 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.362291    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-podinfo\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.362458 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.362352    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-workload-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.362458 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.362379    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-envoy\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.362458 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.362434    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-workload-certs\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.362680 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.362465    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-credential-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.362680 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.362529    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-data\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.362680 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.362568    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istiod-ca-cert\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.362680 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.362598    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4tmv\" (UniqueName: \"kubernetes.io/projected/b460b90d-0d55-44af-a4b9-2e70b3a7b679-kube-api-access-f4tmv\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.362936 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.362909    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-workload-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.362936 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.362923    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-credential-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.363057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.363043    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-workload-certs\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.363258 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.363235    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-data\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.363562 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.363541    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istiod-ca-cert\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.365294 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.365270    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-envoy\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.365404 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.365389    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-podinfo\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.385224 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.385192    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-token\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.385378 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.385310    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4tmv\" (UniqueName: \"kubernetes.io/projected/b460b90d-0d55-44af-a4b9-2e70b3a7b679-kube-api-access-f4tmv\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.532576 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.532491    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:12.843560 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.843479    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"]
Oct 30 00:55:12.847957 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:55:12.847930    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb460b90d_0d55_44af_a4b9_2e70b3a7b679.slice/crio-e54ac2f1eb406863c54f551cfa5649b6e39a83ec02a36d3ec5e459628d67bf62 WatchSource:0}: Error finding container e54ac2f1eb406863c54f551cfa5649b6e39a83ec02a36d3ec5e459628d67bf62: Status 404 returned error can't find the container with id e54ac2f1eb406863c54f551cfa5649b6e39a83ec02a36d3ec5e459628d67bf62
Oct 30 00:55:12.850639 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.850605    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:55:12.850723 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.850672    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:55:12.850723 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:12.850699    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:55:13.793893 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:13.793857    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d" event={"ID":"b460b90d-0d55-44af-a4b9-2e70b3a7b679","Type":"ContainerStarted","Data":"65b1a10ab81c4ae278a3dbb13c7a38f6ac885ce43be98ada7c28696af5455ad3"}
Oct 30 00:55:13.794335 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:13.793914    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d" event={"ID":"b460b90d-0d55-44af-a4b9-2e70b3a7b679","Type":"ContainerStarted","Data":"e54ac2f1eb406863c54f551cfa5649b6e39a83ec02a36d3ec5e459628d67bf62"}
Oct 30 00:55:13.845691 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:13.845635    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d" podStartSLOduration=1.845617232 podStartE2EDuration="1.845617232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:13.844345363 +0000 UTC m=+826.182447275" watchObservedRunningTime="2026-10-30 00:55:13.845617232 +0000 UTC m=+826.183719143"
Oct 30 00:55:14.532917 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:14.532876    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:14.534579 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:14.534554    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:55:14.534702 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:14.534602    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d" podUID="b460b90d-0d55-44af-a4b9-2e70b3a7b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 00:55:15.533212 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:15.533167    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:55:15.533587 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:15.533228    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d" podUID="b460b90d-0d55-44af-a4b9-2e70b3a7b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 00:55:16.533393 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:16.533356    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:55:16.533799 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:16.533410    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d" podUID="b460b90d-0d55-44af-a4b9-2e70b3a7b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 00:55:17.532881 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:17.532849    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:55:17.533063 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:17.532912    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d" podUID="b460b90d-0d55-44af-a4b9-2e70b3a7b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 00:55:18.533390 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:18.533351    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:55:18.533794 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:18.533408    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d" podUID="b460b90d-0d55-44af-a4b9-2e70b3a7b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 00:55:19.040851 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:55:19.040817    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:55:19.540564 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:19.540526    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:19.541018 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:19.540993    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:19.541874 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:55:19.541855    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 00:55:49.040186 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:55:49.040107    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:56:19.039980 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:56:19.039932    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:56:27.884411 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:27.884384    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 00:56:27.884931 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:27.884543    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 00:56:32.333281 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:32.333242    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"]
Oct 30 00:56:32.337879 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:32.337857    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:56:32.363918 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:32.363890    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-39b241\"/\"e2e-conflict-a-988896f8-kserve-self-signed-certs\""
Oct 30 00:56:32.366324 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:32.366294    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"]
Oct 30 00:56:32.425386 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:32.425356    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-model-cache\") pod \"e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9\" (UID: \"4f626f34-1f02-4374-b3dd-33dc3e277540\") " pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:56:32.425386 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:32.425389    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-tmp-dir\") pod \"e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9\" (UID: \"4f626f34-1f02-4374-b3dd-33dc3e277540\") " pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:56:32.425578 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:32.425405    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-home\") pod \"e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9\" (UID: \"4f626f34-1f02-4374-b3dd-33dc3e277540\") " pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:56:32.425578 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:32.425475    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-dshm\") pod \"e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9\" (UID: \"4f626f34-1f02-4374-b3dd-33dc3e277540\") " pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:56:32.425578 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:32.425514    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbllt\" (UniqueName: \"kubernetes.io/projected/4f626f34-1f02-4374-b3dd-33dc3e277540-kube-api-access-vbllt\") pod \"e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9\" (UID: \"4f626f34-1f02-4374-b3dd-33dc3e277540\") " pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:56:32.425735 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:32.425643    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f626f34-1f02-4374-b3dd-33dc3e277540-tls-certs\") pod \"e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9\" (UID: \"4f626f34-1f02-4374-b3dd-33dc3e277540\") " pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:56:32.526500 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:32.526459    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-model-cache\") pod \"e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9\" (UID: \"4f626f34-1f02-4374-b3dd-33dc3e277540\") " pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:56:32.530126 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:32.527149    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-tmp-dir\") pod \"e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9\" (UID: \"4f626f34-1f02-4374-b3dd-33dc3e277540\") " pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:56:32.530126 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:32.527194    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-home\") pod \"e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9\" (UID: \"4f626f34-1f02-4374-b3dd-33dc3e277540\") " pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:56:32.530126 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:32.527220    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-dshm\") pod \"e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9\" (UID: \"4f626f34-1f02-4374-b3dd-33dc3e277540\") " pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:56:32.530126 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:32.527262    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbllt\" (UniqueName: \"kubernetes.io/projected/4f626f34-1f02-4374-b3dd-33dc3e277540-kube-api-access-vbllt\") pod \"e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9\" (UID: \"4f626f34-1f02-4374-b3dd-33dc3e277540\") " pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:56:32.530126 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:32.527301    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f626f34-1f02-4374-b3dd-33dc3e277540-tls-certs\") pod \"e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9\" (UID: \"4f626f34-1f02-4374-b3dd-33dc3e277540\") " pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:56:32.530126 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:32.527637    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-tmp-dir\") pod \"e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9\" (UID: \"4f626f34-1f02-4374-b3dd-33dc3e277540\") " pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:56:32.530126 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:32.526935    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-model-cache\") pod \"e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9\" (UID: \"4f626f34-1f02-4374-b3dd-33dc3e277540\") " pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:56:32.530126 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:32.528086    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-home\") pod \"e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9\" (UID: \"4f626f34-1f02-4374-b3dd-33dc3e277540\") " pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:56:32.531023 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:32.530996    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-dshm\") pod \"e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9\" (UID: \"4f626f34-1f02-4374-b3dd-33dc3e277540\") " pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:56:32.531559 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:32.531456    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f626f34-1f02-4374-b3dd-33dc3e277540-tls-certs\") pod \"e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9\" (UID: \"4f626f34-1f02-4374-b3dd-33dc3e277540\") " pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:56:32.549883 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:32.549857    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbllt\" (UniqueName: \"kubernetes.io/projected/4f626f34-1f02-4374-b3dd-33dc3e277540-kube-api-access-vbllt\") pod \"e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9\" (UID: \"4f626f34-1f02-4374-b3dd-33dc3e277540\") " pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:56:32.655161 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:32.655082    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:56:32.951428 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:32.951393    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"]
Oct 30 00:56:32.954983 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:56:32.954958    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f626f34_1f02_4374_b3dd_33dc3e277540.slice/crio-f757bbe4bfbec70d6259aeedda4f0163e2e5cf19ef135adc8dbd429133e4ead4 WatchSource:0}: Error finding container f757bbe4bfbec70d6259aeedda4f0163e2e5cf19ef135adc8dbd429133e4ead4: Status 404 returned error can't find the container with id f757bbe4bfbec70d6259aeedda4f0163e2e5cf19ef135adc8dbd429133e4ead4
Oct 30 00:56:33.229364 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:33.229278    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9" event={"ID":"4f626f34-1f02-4374-b3dd-33dc3e277540","Type":"ContainerStarted","Data":"734d0dc85baf98379b1e5c91f51609d7222e3219c94c6b69d936c559b45ff4e4"}
Oct 30 00:56:33.229364 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:33.229328    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9" event={"ID":"4f626f34-1f02-4374-b3dd-33dc3e277540","Type":"ContainerStarted","Data":"f757bbe4bfbec70d6259aeedda4f0163e2e5cf19ef135adc8dbd429133e4ead4"}
Oct 30 00:56:33.267225 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:33.267171    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9" podStartSLOduration=1.2671564260000001 podStartE2EDuration="1.267156426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:33.266122011 +0000 UTC m=+905.604223918" watchObservedRunningTime="2026-10-30 00:56:33.267156426 +0000 UTC m=+905.605258338"
Oct 30 00:56:33.655903 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:33.655830    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:56:42.656116 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:42.656084    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:56:42.668783 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:42.668753    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:56:43.297509 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:43.297480    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:56:48.949908 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:48.949876    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"]
Oct 30 00:56:48.954431 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:48.954405    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:56:48.977503 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:48.977477    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-72ed13\"/\"kube-root-ca.crt\""
Oct 30 00:56:48.977657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:48.977505    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-72ed13\"/\"openshift-service-ca.crt\""
Oct 30 00:56:48.977657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:48.977505    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-72ed13\"/\"default-dockercfg-8zpfr\""
Oct 30 00:56:48.977657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:48.977593    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-72ed13\"/\"distinct-w2-72ed13-kserve-self-signed-certs\""
Oct 30 00:56:48.981556 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:48.981532    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"]
Oct 30 00:56:49.039528 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:56:49.039495    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:56:49.102272 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:49.102240    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-tmp-dir\") pod \"distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29\" (UID: \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:56:49.102272 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:49.102274    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-home\") pod \"distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29\" (UID: \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:56:49.102509 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:49.102308    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jvgd\" (UniqueName: \"kubernetes.io/projected/e0ab9bcb-f48a-404b-b787-9f8c43b58722-kube-api-access-6jvgd\") pod \"distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29\" (UID: \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:56:49.102509 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:49.102385    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-dshm\") pod \"distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29\" (UID: \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:56:49.102509 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:49.102476    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ab9bcb-f48a-404b-b787-9f8c43b58722-tls-certs\") pod \"distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29\" (UID: \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:56:49.102632 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:49.102590    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-model-cache\") pod \"distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29\" (UID: \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:56:49.203952 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:49.203922    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jvgd\" (UniqueName: \"kubernetes.io/projected/e0ab9bcb-f48a-404b-b787-9f8c43b58722-kube-api-access-6jvgd\") pod \"distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29\" (UID: \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:56:49.204163 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:49.203970    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-dshm\") pod \"distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29\" (UID: \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:56:49.204163 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:49.204058    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ab9bcb-f48a-404b-b787-9f8c43b58722-tls-certs\") pod \"distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29\" (UID: \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:56:49.204163 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:49.204137    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-model-cache\") pod \"distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29\" (UID: \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:56:49.204331 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:49.204170    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-tmp-dir\") pod \"distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29\" (UID: \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:56:49.204331 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:49.204187    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-home\") pod \"distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29\" (UID: \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:56:49.204605 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:49.204575    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-model-cache\") pod \"distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29\" (UID: \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:56:49.204690 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:49.204570    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-tmp-dir\") pod \"distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29\" (UID: \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:56:49.204690 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:49.204626    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-home\") pod \"distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29\" (UID: \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:56:49.206412 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:49.206388    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-dshm\") pod \"distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29\" (UID: \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:56:49.206721 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:49.206694    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ab9bcb-f48a-404b-b787-9f8c43b58722-tls-certs\") pod \"distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29\" (UID: \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:56:49.225391 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:49.225361    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jvgd\" (UniqueName: \"kubernetes.io/projected/e0ab9bcb-f48a-404b-b787-9f8c43b58722-kube-api-access-6jvgd\") pod \"distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29\" (UID: \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:56:49.269022 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:49.268991    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:56:49.569883 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:49.569804    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"]
Oct 30 00:56:49.573024 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:56:49.572998    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0ab9bcb_f48a_404b_b787_9f8c43b58722.slice/crio-5131dee2fd77ada61c6f20db470ba46b7c6ccef1be2d5770c08348eff14ee79e WatchSource:0}: Error finding container 5131dee2fd77ada61c6f20db470ba46b7c6ccef1be2d5770c08348eff14ee79e: Status 404 returned error can't find the container with id 5131dee2fd77ada61c6f20db470ba46b7c6ccef1be2d5770c08348eff14ee79e
Oct 30 00:56:50.325563 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:50.325523    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29" event={"ID":"e0ab9bcb-f48a-404b-b787-9f8c43b58722","Type":"ContainerStarted","Data":"25fb162a342fdc5a92afe2f784dfc6265c1ee913d8bc226a9b302e4ff3c1c7e0"}
Oct 30 00:56:50.325563 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:50.325571    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29" event={"ID":"e0ab9bcb-f48a-404b-b787-9f8c43b58722","Type":"ContainerStarted","Data":"5131dee2fd77ada61c6f20db470ba46b7c6ccef1be2d5770c08348eff14ee79e"}
Oct 30 00:56:50.366789 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:50.366735    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29" podStartSLOduration=2.366720721 podStartE2EDuration="2.366720721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:50.365186823 +0000 UTC m=+922.703288738" watchObservedRunningTime="2026-10-30 00:56:50.366720721 +0000 UTC m=+922.704822630"
Oct 30 00:56:51.270134 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:51.270101    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:56:59.270055 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:59.270010    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:56:59.283033 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:59.283005    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:56:59.389014 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:56:59.388982    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:57:13.361338 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:13.361252    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"]
Oct 30 00:57:13.366470 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:13.366447    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:57:13.391863 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:13.391822    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d5b2b8\"/\"distinct-w4-d5b2b8-kserve-self-signed-certs\""
Oct 30 00:57:13.394348 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:13.394319    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"]
Oct 30 00:57:13.475649 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:13.475617    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-dshm\") pod \"distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll\" (UID: \"2388a847-21ff-46dd-8b45-4183a2448cfe\") " pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:57:13.475845 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:13.475653    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2388a847-21ff-46dd-8b45-4183a2448cfe-tls-certs\") pod \"distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll\" (UID: \"2388a847-21ff-46dd-8b45-4183a2448cfe\") " pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:57:13.475845 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:13.475797    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-home\") pod \"distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll\" (UID: \"2388a847-21ff-46dd-8b45-4183a2448cfe\") " pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:57:13.475845 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:13.475837    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-tmp-dir\") pod \"distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll\" (UID: \"2388a847-21ff-46dd-8b45-4183a2448cfe\") " pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:57:13.476115 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:13.475994    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvjmn\" (UniqueName: \"kubernetes.io/projected/2388a847-21ff-46dd-8b45-4183a2448cfe-kube-api-access-hvjmn\") pod \"distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll\" (UID: \"2388a847-21ff-46dd-8b45-4183a2448cfe\") " pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:57:13.476115 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:13.476052    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-model-cache\") pod \"distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll\" (UID: \"2388a847-21ff-46dd-8b45-4183a2448cfe\") " pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:57:13.585163 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:13.585117    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-dshm\") pod \"distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll\" (UID: \"2388a847-21ff-46dd-8b45-4183a2448cfe\") " pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:57:13.585163 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:13.585178    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2388a847-21ff-46dd-8b45-4183a2448cfe-tls-certs\") pod \"distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll\" (UID: \"2388a847-21ff-46dd-8b45-4183a2448cfe\") " pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:57:13.585593 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:13.585234    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-home\") pod \"distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll\" (UID: \"2388a847-21ff-46dd-8b45-4183a2448cfe\") " pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:57:13.585593 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:13.585262    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-tmp-dir\") pod \"distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll\" (UID: \"2388a847-21ff-46dd-8b45-4183a2448cfe\") " pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:57:13.585593 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:13.585402    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvjmn\" (UniqueName: \"kubernetes.io/projected/2388a847-21ff-46dd-8b45-4183a2448cfe-kube-api-access-hvjmn\") pod \"distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll\" (UID: \"2388a847-21ff-46dd-8b45-4183a2448cfe\") " pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:57:13.585593 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:13.585455    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-model-cache\") pod \"distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll\" (UID: \"2388a847-21ff-46dd-8b45-4183a2448cfe\") " pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:57:13.586023 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:13.585932    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-model-cache\") pod \"distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll\" (UID: \"2388a847-21ff-46dd-8b45-4183a2448cfe\") " pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:57:13.586713 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:13.586649    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-home\") pod \"distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll\" (UID: \"2388a847-21ff-46dd-8b45-4183a2448cfe\") " pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:57:13.590940 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:13.590909    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-tmp-dir\") pod \"distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll\" (UID: \"2388a847-21ff-46dd-8b45-4183a2448cfe\") " pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:57:13.591946 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:13.591917    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2388a847-21ff-46dd-8b45-4183a2448cfe-tls-certs\") pod \"distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll\" (UID: \"2388a847-21ff-46dd-8b45-4183a2448cfe\") " pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:57:13.592211 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:13.592183    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-dshm\") pod \"distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll\" (UID: \"2388a847-21ff-46dd-8b45-4183a2448cfe\") " pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:57:13.608222 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:13.608188    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvjmn\" (UniqueName: \"kubernetes.io/projected/2388a847-21ff-46dd-8b45-4183a2448cfe-kube-api-access-hvjmn\") pod \"distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll\" (UID: \"2388a847-21ff-46dd-8b45-4183a2448cfe\") " pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:57:13.683338 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:13.683249    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:57:13.981273 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:13.981239    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"]
Oct 30 00:57:13.984878 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:57:13.984853    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2388a847_21ff_46dd_8b45_4183a2448cfe.slice/crio-c168ad2e8f2534bb42969b9e7e57ce971214ec67f63d9a94748d4bebf6e5fecb WatchSource:0}: Error finding container c168ad2e8f2534bb42969b9e7e57ce971214ec67f63d9a94748d4bebf6e5fecb: Status 404 returned error can't find the container with id c168ad2e8f2534bb42969b9e7e57ce971214ec67f63d9a94748d4bebf6e5fecb
Oct 30 00:57:14.462011 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:14.461973    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll" event={"ID":"2388a847-21ff-46dd-8b45-4183a2448cfe","Type":"ContainerStarted","Data":"5a8d3d1fed736a70ceabb889c1b3bf6c6e1af01cbc8dba83e609cfff8d6f9234"}
Oct 30 00:57:14.462435 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:14.462020    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll" event={"ID":"2388a847-21ff-46dd-8b45-4183a2448cfe","Type":"ContainerStarted","Data":"c168ad2e8f2534bb42969b9e7e57ce971214ec67f63d9a94748d4bebf6e5fecb"}
Oct 30 00:57:14.501053 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:14.501001    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll" podStartSLOduration=1.50098745 podStartE2EDuration="1.50098745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:14.499352551 +0000 UTC m=+946.837454453" watchObservedRunningTime="2026-10-30 00:57:14.50098745 +0000 UTC m=+946.839089360"
Oct 30 00:57:14.683559 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:14.683527    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:57:19.042500 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:57:19.042464    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:57:20.298359 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.298325    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"]
Oct 30 00:57:20.302380 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.302357    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.317667 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.317643    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9bfe36-openshift-default-dockercfg-4n5g9\""
Oct 30 00:57:20.330603 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.330578    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"]
Oct 30 00:57:20.354055 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.353994    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8150dfe-249d-4cbe-9652-226dd18509f5-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.354055 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.354040    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.354249 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.354117    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.354249 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.354166    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.354249 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.354236    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.354375 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.354263    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntgtb\" (UniqueName: \"kubernetes.io/projected/a8150dfe-249d-4cbe-9652-226dd18509f5-kube-api-access-ntgtb\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.354375 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.354300    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.354457 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.354433    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.354498 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.354479    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.455839 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.455807    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.456002 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.455926    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.456002 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.455979    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.456116 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.456028    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8150dfe-249d-4cbe-9652-226dd18509f5-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.456116 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.456062    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.456236 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.456115    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.456236 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.456163    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.456236 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.456197    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.456236 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.456227    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntgtb\" (UniqueName: \"kubernetes.io/projected/a8150dfe-249d-4cbe-9652-226dd18509f5-kube-api-access-ntgtb\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.456430 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.456319    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.456430 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.456373    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.456689 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.456663    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8150dfe-249d-4cbe-9652-226dd18509f5-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.456786 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.456761    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.456832 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.456804    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.458497 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.458472    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.458860 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.458839    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.477631 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.477596    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.477803 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.477787    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntgtb\" (UniqueName: \"kubernetes.io/projected/a8150dfe-249d-4cbe-9652-226dd18509f5-kube-api-access-ntgtb\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.615617 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.615542    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:20.928240 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.928167    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"]
Oct 30 00:57:20.931654 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:57:20.931628    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8150dfe_249d_4cbe_9652_226dd18509f5.slice/crio-7e9574ffc042fd32bd92aa085d0239a6750c4e68467b8ca5110eb7c965115a98 WatchSource:0}: Error finding container 7e9574ffc042fd32bd92aa085d0239a6750c4e68467b8ca5110eb7c965115a98: Status 404 returned error can't find the container with id 7e9574ffc042fd32bd92aa085d0239a6750c4e68467b8ca5110eb7c965115a98
Oct 30 00:57:20.934198 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.934169    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:57:20.934265 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.934241    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:57:20.934320 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:20.934299    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:57:21.502088 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:21.502044    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt" event={"ID":"a8150dfe-249d-4cbe-9652-226dd18509f5","Type":"ContainerStarted","Data":"67dddb7f344b9aa5de80dc5c7a287ea70541a106bc9fee0090cd52cbbb985ef7"}
Oct 30 00:57:21.502533 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:21.502121    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt" event={"ID":"a8150dfe-249d-4cbe-9652-226dd18509f5","Type":"ContainerStarted","Data":"7e9574ffc042fd32bd92aa085d0239a6750c4e68467b8ca5110eb7c965115a98"}
Oct 30 00:57:21.579097 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:21.556045    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt" podStartSLOduration=1.556030381 podStartE2EDuration="1.556030381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:21.553945357 +0000 UTC m=+953.892047293" watchObservedRunningTime="2026-10-30 00:57:21.556030381 +0000 UTC m=+953.894132291"
Oct 30 00:57:21.616118 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:21.616058    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:21.617408 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:21.617384    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:57:21.617541 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:21.617436    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt" podUID="a8150dfe-249d-4cbe-9652-226dd18509f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 00:57:22.616026 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:22.615996    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:57:22.616481 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:22.616043    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt" podUID="a8150dfe-249d-4cbe-9652-226dd18509f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 00:57:23.615993 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:23.615960    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:57:23.616179 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:23.616008    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt" podUID="a8150dfe-249d-4cbe-9652-226dd18509f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 00:57:23.684383 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:23.684341    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:57:23.696917 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:23.696889    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:57:24.528356 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:24.528322    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:57:24.616373 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:24.616335    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:57:24.616739 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:24.616389    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt" podUID="a8150dfe-249d-4cbe-9652-226dd18509f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 00:57:25.616832 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:25.616797    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:57:25.617259 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:25.616850    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt" podUID="a8150dfe-249d-4cbe-9652-226dd18509f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 00:57:25.932126 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:25.932019    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"]
Oct 30 00:57:25.936952 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:25.936927    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:57:25.988466 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:25.988435    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-72ed13\"/\"distinct-2-w2-72ed13-kserve-self-signed-certs\""
Oct 30 00:57:25.991330 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:25.991303    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"]
Oct 30 00:57:26.029711 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:26.029677    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-tmp-dir\") pod \"distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz\" (UID: \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:57:26.029903 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:26.029743    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-tls-certs\") pod \"distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz\" (UID: \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:57:26.029903 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:26.029836    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-model-cache\") pod \"distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz\" (UID: \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:57:26.029903 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:26.029874    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-dshm\") pod \"distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz\" (UID: \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:57:26.030105 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:26.030005    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-home\") pod \"distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz\" (UID: \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:57:26.030105 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:26.030040    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8dfm\" (UniqueName: \"kubernetes.io/projected/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-kube-api-access-x8dfm\") pod \"distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz\" (UID: \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:57:26.131290 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:26.131258    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8dfm\" (UniqueName: \"kubernetes.io/projected/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-kube-api-access-x8dfm\") pod \"distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz\" (UID: \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:57:26.131469 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:26.131321    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-tmp-dir\") pod \"distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz\" (UID: \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:57:26.131469 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:26.131374    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-tls-certs\") pod \"distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz\" (UID: \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:57:26.131469 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:26.131439    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-model-cache\") pod \"distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz\" (UID: \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:57:26.131638 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:26.131611    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-dshm\") pod \"distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz\" (UID: \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:57:26.131745 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:26.131725    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-home\") pod \"distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz\" (UID: \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:57:26.131827 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:26.131803    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-tmp-dir\") pod \"distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz\" (UID: \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:57:26.131886 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:26.131858    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-model-cache\") pod \"distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz\" (UID: \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:57:26.132030 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:26.132013    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-home\") pod \"distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz\" (UID: \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:57:26.133900 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:26.133878    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-dshm\") pod \"distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz\" (UID: \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:57:26.134557 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:26.134535    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-tls-certs\") pod \"distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz\" (UID: \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:57:26.152575 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:26.152552    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8dfm\" (UniqueName: \"kubernetes.io/projected/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-kube-api-access-x8dfm\") pod \"distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz\" (UID: \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\") " pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:57:26.250223 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:26.250180    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:57:26.616247 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:26.616160    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:57:26.616247 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:26.616224    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt" podUID="a8150dfe-249d-4cbe-9652-226dd18509f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 00:57:26.617991 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:26.617964    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"]
Oct 30 00:57:26.621970 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:57:26.621943    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86e98e6a_b1cf_4d08_b807_26397cd1b4a4.slice/crio-a9d504d64e9c28ec0e3fade134f084be2bcb5591fdb1e6cf43cd00b18028db6c WatchSource:0}: Error finding container a9d504d64e9c28ec0e3fade134f084be2bcb5591fdb1e6cf43cd00b18028db6c: Status 404 returned error can't find the container with id a9d504d64e9c28ec0e3fade134f084be2bcb5591fdb1e6cf43cd00b18028db6c
Oct 30 00:57:27.536933 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.536891    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz" event={"ID":"86e98e6a-b1cf-4d08-b807-26397cd1b4a4","Type":"ContainerStarted","Data":"de65ecc9869bae257c2ab7b41545bd61b89f9e541d446d1e8fec352138b61dc5"}
Oct 30 00:57:27.537257 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.537220    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz" event={"ID":"86e98e6a-b1cf-4d08-b807-26397cd1b4a4","Type":"ContainerStarted","Data":"a9d504d64e9c28ec0e3fade134f084be2bcb5591fdb1e6cf43cd00b18028db6c"}
Oct 30 00:57:27.578038 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.577971    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz" podStartSLOduration=2.577953794 podStartE2EDuration="2.577953794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:27.574352087 +0000 UTC m=+959.912453997" watchObservedRunningTime="2026-10-30 00:57:27.577953794 +0000 UTC m=+959.916055707"
Oct 30 00:57:27.596919 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.596891    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"]
Oct 30 00:57:27.600838 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.600818    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.624404 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.624375    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"]
Oct 30 00:57:27.652429 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.652396    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-data\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.652595 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.652561    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-podinfo\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.652665 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.652610    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-envoy\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.652665 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.652639    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istiod-ca-cert\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.652775 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.652746    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-credential-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.652846 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.652798    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-workload-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.652846 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.652825    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-token\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.652949 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.652890    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j45z8\" (UniqueName: \"kubernetes.io/projected/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-kube-api-access-j45z8\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.653006 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.652992    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-workload-certs\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.754021 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.753986    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-workload-certs\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.754207 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.754041    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-data\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.754207 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.754113    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-podinfo\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.754207 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.754141    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-envoy\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.754207 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.754165    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istiod-ca-cert\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.754444 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.754224    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-credential-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.754444 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.754258    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-workload-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.754444 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.754280    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-token\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.754444 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.754338    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j45z8\" (UniqueName: \"kubernetes.io/projected/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-kube-api-access-j45z8\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.754646 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.754506    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-workload-certs\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.754646 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.754539    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-data\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.754806 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.754785    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-credential-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.754882 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.754815    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-workload-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.755295 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.755273    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istiod-ca-cert\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.757140 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.757116    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-podinfo\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.757255 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.757120    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-envoy\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.776353 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.776321    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-token\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.776999 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.776974    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j45z8\" (UniqueName: \"kubernetes.io/projected/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-kube-api-access-j45z8\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.915125 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.914984    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:27.919083 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.919036    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:27.919494 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.919468    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:27.921557 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:27.921529    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 00:57:28.231775 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:28.231735    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"]
Oct 30 00:57:28.235629 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:57:28.235604    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64b0b95_75f6_4d2a_9a26_dc73d2c53ea6.slice/crio-a5c3c1d7cdeadcbdc5c47baaa6ac997ccfdf205f2b96dea338cb6be9e15ff0c4 WatchSource:0}: Error finding container a5c3c1d7cdeadcbdc5c47baaa6ac997ccfdf205f2b96dea338cb6be9e15ff0c4: Status 404 returned error can't find the container with id a5c3c1d7cdeadcbdc5c47baaa6ac997ccfdf205f2b96dea338cb6be9e15ff0c4
Oct 30 00:57:28.238084 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:28.238036    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:57:28.238170 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:28.238131    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:57:28.238244 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:28.238181    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:57:28.251084 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:28.251050    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:57:28.544976 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:28.544878    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g" event={"ID":"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6","Type":"ContainerStarted","Data":"be243ed6cca81820e78ead50eedf580c7d4d595657fe9e914705ddaf37306ba9"}
Oct 30 00:57:28.544976 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:28.544945    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g" event={"ID":"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6","Type":"ContainerStarted","Data":"a5c3c1d7cdeadcbdc5c47baaa6ac997ccfdf205f2b96dea338cb6be9e15ff0c4"}
Oct 30 00:57:28.589788 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:28.589731    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g" podStartSLOduration=1.589713013 podStartE2EDuration="1.589713013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:28.586372496 +0000 UTC m=+960.924474405" watchObservedRunningTime="2026-10-30 00:57:28.589713013 +0000 UTC m=+960.927814924"
Oct 30 00:57:28.915553 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:28.915471    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:29.916982 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:29.916942    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 00:57:29.917402 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:29.916998    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g" podUID="e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 00:57:30.915832 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:30.915788    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 00:57:30.916114 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:30.915845    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g" podUID="e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 00:57:31.916246 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:31.916208    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 00:57:31.916646 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:31.916270    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g" podUID="e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 00:57:32.916155 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:32.916117    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 00:57:32.916340 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:32.916171    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g" podUID="e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 00:57:33.915847 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:33.915809    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 00:57:33.916019 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:33.915872    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g" podUID="e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 00:57:35.198740 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:35.198705    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:35.199657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:35.199626    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:35.202258 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:35.202236    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 00:57:36.250634 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:36.250594    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:57:36.263395 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:36.263370    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:57:36.623137 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:36.623036    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:57:47.682009 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:57:47.681971    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"]
Oct 30 00:57:49.043232 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:57:49.043193    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:58:02.705814 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:02.705754    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9" podUID="4f626f34-1f02-4374-b3dd-33dc3e277540" containerName="main" containerID="cri-o://734d0dc85baf98379b1e5c91f51609d7222e3219c94c6b69d936c559b45ff4e4" gracePeriod=45
Oct 30 00:58:02.957159 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:02.957098    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:58:03.086701 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.086674    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-model-cache\") pod \"4f626f34-1f02-4374-b3dd-33dc3e277540\" (UID: \"4f626f34-1f02-4374-b3dd-33dc3e277540\") "
Oct 30 00:58:03.086888 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.086766    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-tmp-dir\") pod \"4f626f34-1f02-4374-b3dd-33dc3e277540\" (UID: \"4f626f34-1f02-4374-b3dd-33dc3e277540\") "
Oct 30 00:58:03.086888 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.086819    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-dshm\") pod \"4f626f34-1f02-4374-b3dd-33dc3e277540\" (UID: \"4f626f34-1f02-4374-b3dd-33dc3e277540\") "
Oct 30 00:58:03.086888 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.086863    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f626f34-1f02-4374-b3dd-33dc3e277540-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f626f34-1f02-4374-b3dd-33dc3e277540-tls-certs\") pod \"4f626f34-1f02-4374-b3dd-33dc3e277540\" (UID: \"4f626f34-1f02-4374-b3dd-33dc3e277540\") "
Oct 30 00:58:03.087092 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.086927    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-model-cache" pod "4f626f34-1f02-4374-b3dd-33dc3e277540" (UID: "4f626f34-1f02-4374-b3dd-33dc3e277540"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:03.087092 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.086942    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f626f34-1f02-4374-b3dd-33dc3e277540-kube-api-access-vbllt\" (UniqueName: \"kubernetes.io/projected/4f626f34-1f02-4374-b3dd-33dc3e277540-kube-api-access-vbllt\") pod \"4f626f34-1f02-4374-b3dd-33dc3e277540\" (UID: \"4f626f34-1f02-4374-b3dd-33dc3e277540\") "
Oct 30 00:58:03.087092 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.087022    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-home\" (UniqueName: \"kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-home\") pod \"4f626f34-1f02-4374-b3dd-33dc3e277540\" (UID: \"4f626f34-1f02-4374-b3dd-33dc3e277540\") "
Oct 30 00:58:03.087092 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.087048    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-tmp-dir" pod "4f626f34-1f02-4374-b3dd-33dc3e277540" (UID: "4f626f34-1f02-4374-b3dd-33dc3e277540"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:03.087312 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.087288    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-home" pod "4f626f34-1f02-4374-b3dd-33dc3e277540" (UID: "4f626f34-1f02-4374-b3dd-33dc3e277540"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:03.087481 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.087458    4864 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-home\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:58:03.087481 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.087482    4864 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-model-cache\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:58:03.087641 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.087518    4864 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-tmp-dir\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:58:03.089308 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.089277    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-dshm" pod "4f626f34-1f02-4374-b3dd-33dc3e277540" (UID: "4f626f34-1f02-4374-b3dd-33dc3e277540"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:03.089308 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.089289    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f626f34-1f02-4374-b3dd-33dc3e277540-tls-certs" pod "4f626f34-1f02-4374-b3dd-33dc3e277540" (UID: "4f626f34-1f02-4374-b3dd-33dc3e277540"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:58:03.089421 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.089335    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f626f34-1f02-4374-b3dd-33dc3e277540-kube-api-access-vbllt" pod "4f626f34-1f02-4374-b3dd-33dc3e277540" (UID: "4f626f34-1f02-4374-b3dd-33dc3e277540"). InnerVolumeSpecName "kube-api-access-vbllt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:03.188861 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.188830    4864 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f626f34-1f02-4374-b3dd-33dc3e277540-tls-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:58:03.188861 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.188857    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbllt\" (UniqueName: \"kubernetes.io/projected/4f626f34-1f02-4374-b3dd-33dc3e277540-kube-api-access-vbllt\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:58:03.188861 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.188870    4864 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f626f34-1f02-4374-b3dd-33dc3e277540-dshm\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:58:03.307936 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.307856    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"]
Oct 30 00:58:03.308490 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.308471    4864 cpu_manager.go:414] "containerMap: removing container" podUID="4f626f34-1f02-4374-b3dd-33dc3e277540" containerName="main"
Oct 30 00:58:03.308490 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.308490    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f626f34-1f02-4374-b3dd-33dc3e277540" containerName="main"
Oct 30 00:58:03.308607 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.308568    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f626f34-1f02-4374-b3dd-33dc3e277540" containerName="main"
Oct 30 00:58:03.312329 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.312311    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.340760 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.340730    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"]
Oct 30 00:58:03.391173 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.391145    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-istio-envoy\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.391335 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.391209    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-istio-data\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.391335 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.391305    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e74574-f055-44f7-b920-ef38942e05c5-istio-token\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.391440 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.391355    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-workload-certs\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.391503 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.391451    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-credential-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.391562 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.391497    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e74574-f055-44f7-b920-ef38942e05c5-istiod-ca-cert\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.391562 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.391552    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-workload-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.391643 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.391619    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e74574-f055-44f7-b920-ef38942e05c5-istio-podinfo\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.391693 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.391679    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkdp5\" (UniqueName: \"kubernetes.io/projected/b2e74574-f055-44f7-b920-ef38942e05c5-kube-api-access-nkdp5\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.492735 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.492697    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e74574-f055-44f7-b920-ef38942e05c5-istio-token\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.492892 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.492759    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-workload-certs\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.492892 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.492782    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-credential-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.492892 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.492806    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e74574-f055-44f7-b920-ef38942e05c5-istiod-ca-cert\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.492892 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.492837    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-workload-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.492892 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.492870    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e74574-f055-44f7-b920-ef38942e05c5-istio-podinfo\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.493234 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.492923    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkdp5\" (UniqueName: \"kubernetes.io/projected/b2e74574-f055-44f7-b920-ef38942e05c5-kube-api-access-nkdp5\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.493234 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.492993    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-istio-envoy\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.493234 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.493041    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-istio-data\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.493389 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.493320    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-workload-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.493460 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.493438    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-workload-certs\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.493548 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.493524    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-istio-data\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.493667 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.493627    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-credential-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.493745 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.493712    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e74574-f055-44f7-b920-ef38942e05c5-istiod-ca-cert\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.495367 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.495347    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-istio-envoy\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.495495 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.495479    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e74574-f055-44f7-b920-ef38942e05c5-istio-podinfo\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.516200 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.516181    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e74574-f055-44f7-b920-ef38942e05c5-istio-token\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.516456 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.516440    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkdp5\" (UniqueName: \"kubernetes.io/projected/b2e74574-f055-44f7-b920-ef38942e05c5-kube-api-access-nkdp5\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.626760 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.626686    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:03.776570 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.776536    4864 generic.go:356] "Generic (PLEG): container finished" podID="4f626f34-1f02-4374-b3dd-33dc3e277540" containerID="734d0dc85baf98379b1e5c91f51609d7222e3219c94c6b69d936c559b45ff4e4" exitCode=0
Oct 30 00:58:03.776996 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.776580    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"
Oct 30 00:58:03.776996 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.776617    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9" event={"ID":"4f626f34-1f02-4374-b3dd-33dc3e277540","Type":"ContainerDied","Data":"734d0dc85baf98379b1e5c91f51609d7222e3219c94c6b69d936c559b45ff4e4"}
Oct 30 00:58:03.776996 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.776663    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9" event={"ID":"4f626f34-1f02-4374-b3dd-33dc3e277540","Type":"ContainerDied","Data":"f757bbe4bfbec70d6259aeedda4f0163e2e5cf19ef135adc8dbd429133e4ead4"}
Oct 30 00:58:03.776996 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.776680    4864 scope.go:122] "RemoveContainer" containerID="734d0dc85baf98379b1e5c91f51609d7222e3219c94c6b69d936c559b45ff4e4"
Oct 30 00:58:03.786475 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.786457    4864 scope.go:122] "RemoveContainer" containerID="734d0dc85baf98379b1e5c91f51609d7222e3219c94c6b69d936c559b45ff4e4"
Oct 30 00:58:03.786774 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:58:03.786754    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"734d0dc85baf98379b1e5c91f51609d7222e3219c94c6b69d936c559b45ff4e4\": container with ID starting with 734d0dc85baf98379b1e5c91f51609d7222e3219c94c6b69d936c559b45ff4e4 not found: ID does not exist" containerID="734d0dc85baf98379b1e5c91f51609d7222e3219c94c6b69d936c559b45ff4e4"
Oct 30 00:58:03.786826 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.786781    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"734d0dc85baf98379b1e5c91f51609d7222e3219c94c6b69d936c559b45ff4e4"} err="failed to get container status \"734d0dc85baf98379b1e5c91f51609d7222e3219c94c6b69d936c559b45ff4e4\": rpc error: code = NotFound desc = could not find container \"734d0dc85baf98379b1e5c91f51609d7222e3219c94c6b69d936c559b45ff4e4\": container with ID starting with 734d0dc85baf98379b1e5c91f51609d7222e3219c94c6b69d936c559b45ff4e4 not found: ID does not exist"
Oct 30 00:58:03.825878 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.825855    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"]
Oct 30 00:58:03.828558 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.828537    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-39b241/e2e-conflict-a-988896f8-kserve-767d8989c9-j2mq9"]
Oct 30 00:58:03.940380 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.940303    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"]
Oct 30 00:58:03.943851 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:58:03.943814    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e74574_f055_44f7_b920_ef38942e05c5.slice/crio-36159c11fc5dc8e89bf26cae0a626dbc4a8861906e955408b364065fbd79f6ea WatchSource:0}: Error finding container 36159c11fc5dc8e89bf26cae0a626dbc4a8861906e955408b364065fbd79f6ea: Status 404 returned error can't find the container with id 36159c11fc5dc8e89bf26cae0a626dbc4a8861906e955408b364065fbd79f6ea
Oct 30 00:58:03.946283 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.946252    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:58:03.946356 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.946315    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:58:03.946356 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:03.946342    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:58:04.792710 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:04.792676    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h" event={"ID":"b2e74574-f055-44f7-b920-ef38942e05c5","Type":"ContainerStarted","Data":"4c549b18a76030ee6b3c03ad2ae308b1e4f643f7e56ffd92929ca3863e864072"}
Oct 30 00:58:04.793220 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:04.792723    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h" event={"ID":"b2e74574-f055-44f7-b920-ef38942e05c5","Type":"ContainerStarted","Data":"36159c11fc5dc8e89bf26cae0a626dbc4a8861906e955408b364065fbd79f6ea"}
Oct 30 00:58:04.892552 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:04.892497    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h" podStartSLOduration=1.892481581 podStartE2EDuration="1.892481581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:04.883590066 +0000 UTC m=+997.221691987" watchObservedRunningTime="2026-10-30 00:58:04.892481581 +0000 UTC m=+997.230583491"
Oct 30 00:58:05.627484 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:05.627435    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:05.629031 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:05.629002    4864 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 00:58:05.629202 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:05.629048    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h" podUID="b2e74574-f055-44f7-b920-ef38942e05c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 00:58:05.793753 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:05.793718    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f626f34-1f02-4374-b3dd-33dc3e277540" path="/var/lib/kubelet/pods/4f626f34-1f02-4374-b3dd-33dc3e277540/volumes"
Oct 30 00:58:06.627671 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:06.627631    4864 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 00:58:06.627855 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:06.627686    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h" podUID="b2e74574-f055-44f7-b920-ef38942e05c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 00:58:06.852704 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:06.852667    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"]
Oct 30 00:58:06.868369 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:06.868338    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:58:06.887509 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:06.887430    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-39b241\"/\"distinct-w3-39b241-kserve-self-signed-certs\""
Oct 30 00:58:06.889843 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:06.889815    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"]
Oct 30 00:58:07.035163 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.035131    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-dshm\") pod \"distinct-w3-39b241-kserve-7bc8898fd9-s5qsj\" (UID: \"63a8525d-9951-4f19-b124-62e4fa91c6b9\") " pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:58:07.035309 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.035173    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-model-cache\") pod \"distinct-w3-39b241-kserve-7bc8898fd9-s5qsj\" (UID: \"63a8525d-9951-4f19-b124-62e4fa91c6b9\") " pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:58:07.035309 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.035196    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4c9c\" (UniqueName: \"kubernetes.io/projected/63a8525d-9951-4f19-b124-62e4fa91c6b9-kube-api-access-j4c9c\") pod \"distinct-w3-39b241-kserve-7bc8898fd9-s5qsj\" (UID: \"63a8525d-9951-4f19-b124-62e4fa91c6b9\") " pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:58:07.035309 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.035293    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-home\") pod \"distinct-w3-39b241-kserve-7bc8898fd9-s5qsj\" (UID: \"63a8525d-9951-4f19-b124-62e4fa91c6b9\") " pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:58:07.035451 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.035310    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63a8525d-9951-4f19-b124-62e4fa91c6b9-tls-certs\") pod \"distinct-w3-39b241-kserve-7bc8898fd9-s5qsj\" (UID: \"63a8525d-9951-4f19-b124-62e4fa91c6b9\") " pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:58:07.035451 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.035373    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-tmp-dir\") pod \"distinct-w3-39b241-kserve-7bc8898fd9-s5qsj\" (UID: \"63a8525d-9951-4f19-b124-62e4fa91c6b9\") " pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:58:07.137053 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.137018    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-dshm\") pod \"distinct-w3-39b241-kserve-7bc8898fd9-s5qsj\" (UID: \"63a8525d-9951-4f19-b124-62e4fa91c6b9\") " pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:58:07.137053 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.137060    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-model-cache\") pod \"distinct-w3-39b241-kserve-7bc8898fd9-s5qsj\" (UID: \"63a8525d-9951-4f19-b124-62e4fa91c6b9\") " pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:58:07.137313 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.137096    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4c9c\" (UniqueName: \"kubernetes.io/projected/63a8525d-9951-4f19-b124-62e4fa91c6b9-kube-api-access-j4c9c\") pod \"distinct-w3-39b241-kserve-7bc8898fd9-s5qsj\" (UID: \"63a8525d-9951-4f19-b124-62e4fa91c6b9\") " pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:58:07.137313 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.137129    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-home\") pod \"distinct-w3-39b241-kserve-7bc8898fd9-s5qsj\" (UID: \"63a8525d-9951-4f19-b124-62e4fa91c6b9\") " pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:58:07.137313 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.137145    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63a8525d-9951-4f19-b124-62e4fa91c6b9-tls-certs\") pod \"distinct-w3-39b241-kserve-7bc8898fd9-s5qsj\" (UID: \"63a8525d-9951-4f19-b124-62e4fa91c6b9\") " pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:58:07.137313 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.137175    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-tmp-dir\") pod \"distinct-w3-39b241-kserve-7bc8898fd9-s5qsj\" (UID: \"63a8525d-9951-4f19-b124-62e4fa91c6b9\") " pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:58:07.137623 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.137571    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-model-cache\") pod \"distinct-w3-39b241-kserve-7bc8898fd9-s5qsj\" (UID: \"63a8525d-9951-4f19-b124-62e4fa91c6b9\") " pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:58:07.137623 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.137611    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-home\") pod \"distinct-w3-39b241-kserve-7bc8898fd9-s5qsj\" (UID: \"63a8525d-9951-4f19-b124-62e4fa91c6b9\") " pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:58:07.137713 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.137633    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-tmp-dir\") pod \"distinct-w3-39b241-kserve-7bc8898fd9-s5qsj\" (UID: \"63a8525d-9951-4f19-b124-62e4fa91c6b9\") " pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:58:07.139403 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.139381    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-dshm\") pod \"distinct-w3-39b241-kserve-7bc8898fd9-s5qsj\" (UID: \"63a8525d-9951-4f19-b124-62e4fa91c6b9\") " pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:58:07.139570 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.139555    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63a8525d-9951-4f19-b124-62e4fa91c6b9-tls-certs\") pod \"distinct-w3-39b241-kserve-7bc8898fd9-s5qsj\" (UID: \"63a8525d-9951-4f19-b124-62e4fa91c6b9\") " pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:58:07.181784 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.181746    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4c9c\" (UniqueName: \"kubernetes.io/projected/63a8525d-9951-4f19-b124-62e4fa91c6b9-kube-api-access-j4c9c\") pod \"distinct-w3-39b241-kserve-7bc8898fd9-s5qsj\" (UID: \"63a8525d-9951-4f19-b124-62e4fa91c6b9\") " pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:58:07.197382 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.197355    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:58:07.497209 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.497175    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"]
Oct 30 00:58:07.500683 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:58:07.500652    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63a8525d_9951_4f19_b124_62e4fa91c6b9.slice/crio-8da544390b0e5b084c3c9055600b68195c46c983d6c7f94f79dc47a47a6f0bb5 WatchSource:0}: Error finding container 8da544390b0e5b084c3c9055600b68195c46c983d6c7f94f79dc47a47a6f0bb5: Status 404 returned error can't find the container with id 8da544390b0e5b084c3c9055600b68195c46c983d6c7f94f79dc47a47a6f0bb5
Oct 30 00:58:07.628112 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.627719    4864 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 00:58:07.628112 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.627781    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h" podUID="b2e74574-f055-44f7-b920-ef38942e05c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 00:58:07.812234 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.812121    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj" event={"ID":"63a8525d-9951-4f19-b124-62e4fa91c6b9","Type":"ContainerStarted","Data":"6502f4b99f6280f96497ac9b1f232328a77327da71dca13f37c8b652b50ec961"}
Oct 30 00:58:07.812234 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.812166    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj" event={"ID":"63a8525d-9951-4f19-b124-62e4fa91c6b9","Type":"ContainerStarted","Data":"8da544390b0e5b084c3c9055600b68195c46c983d6c7f94f79dc47a47a6f0bb5"}
Oct 30 00:58:07.851953 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:07.851881    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj" podStartSLOduration=1.851866971 podStartE2EDuration="1.851866971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:07.849838443 +0000 UTC m=+1000.187940354" watchObservedRunningTime="2026-10-30 00:58:07.851866971 +0000 UTC m=+1000.189968943"
Oct 30 00:58:08.214174 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:08.213474    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:58:08.627353 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:08.627256    4864 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 00:58:08.627627 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:08.627582    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h" podUID="b2e74574-f055-44f7-b920-ef38942e05c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 00:58:09.627863 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:09.627815    4864 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 00:58:09.628297 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:09.627878    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h" podUID="b2e74574-f055-44f7-b920-ef38942e05c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 00:58:10.627760 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:10.627728    4864 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 00:58:10.627936 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:10.627781    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h" podUID="b2e74574-f055-44f7-b920-ef38942e05c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 00:58:12.591951 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:12.591916    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:12.592410 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:12.592331    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:12.593251 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:12.593231    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 00:58:17.198265 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:17.198226    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:58:17.210641 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:17.210618    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:58:17.917528 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:17.917493    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:58:19.039361 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:58:19.039319    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:58:27.803624 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:27.803590    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"]
Oct 30 00:58:27.808391 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:27.808363    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:27.839913 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:27.839881    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"]
Oct 30 00:58:27.950835 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:27.950793    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-workload-certs\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:27.950835 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:27.950829    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76501308-cde8-41f2-aa4e-b6039b294be5-istio-token\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:27.951053 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:27.950853    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmsd7\" (UniqueName: \"kubernetes.io/projected/76501308-cde8-41f2-aa4e-b6039b294be5-kube-api-access-lmsd7\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:27.951053 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:27.950983    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76501308-cde8-41f2-aa4e-b6039b294be5-istio-podinfo\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:27.951053 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:27.951018    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-credential-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:27.951053 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:27.951041    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76501308-cde8-41f2-aa4e-b6039b294be5-istiod-ca-cert\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:27.951224 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:27.951150    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-istio-envoy\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:27.951224 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:27.951173    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-workload-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:27.951309 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:27.951257    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-istio-data\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:28.052561 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:28.052527    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76501308-cde8-41f2-aa4e-b6039b294be5-istio-podinfo\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:28.052561 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:28.052566    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-credential-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:28.052814 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:28.052584    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76501308-cde8-41f2-aa4e-b6039b294be5-istiod-ca-cert\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:28.052814 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:28.052622    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-istio-envoy\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:28.052814 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:28.052641    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-workload-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:28.052814 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:28.052701    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-istio-data\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:28.053016 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:28.052845    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-workload-certs\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:28.053016 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:28.052876    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76501308-cde8-41f2-aa4e-b6039b294be5-istio-token\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:28.053016 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:28.052907    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmsd7\" (UniqueName: \"kubernetes.io/projected/76501308-cde8-41f2-aa4e-b6039b294be5-kube-api-access-lmsd7\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:28.053224 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:28.053204    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-workload-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:28.053288 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:28.053246    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-istio-data\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:28.053288 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:28.053272    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-workload-certs\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:28.053404 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:28.053355    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-credential-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:28.053460 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:28.053429    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76501308-cde8-41f2-aa4e-b6039b294be5-istiod-ca-cert\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:28.055039 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:28.054982    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-istio-envoy\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:28.055251 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:28.055233    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76501308-cde8-41f2-aa4e-b6039b294be5-istio-podinfo\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:28.074124 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:28.074099    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76501308-cde8-41f2-aa4e-b6039b294be5-istio-token\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:28.074586 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:28.074564    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmsd7\" (UniqueName: \"kubernetes.io/projected/76501308-cde8-41f2-aa4e-b6039b294be5-kube-api-access-lmsd7\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:28.126256 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:28.126223    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:28.456448 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:28.456420    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"]
Oct 30 00:58:28.460085 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:58:28.460043    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76501308_cde8_41f2_aa4e_b6039b294be5.slice/crio-5a49ae7540aad559af2607523ea7edcdae0220db820e7294392aa4de33fcdc26 WatchSource:0}: Error finding container 5a49ae7540aad559af2607523ea7edcdae0220db820e7294392aa4de33fcdc26: Status 404 returned error can't find the container with id 5a49ae7540aad559af2607523ea7edcdae0220db820e7294392aa4de33fcdc26
Oct 30 00:58:28.462854 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:28.462824    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:58:28.462937 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:28.462890    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:58:28.462937 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:28.462925    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:58:29.002646 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:29.002611    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62" event={"ID":"76501308-cde8-41f2-aa4e-b6039b294be5","Type":"ContainerStarted","Data":"d8eb9024b6b25e44b25d15bd00c7cf401ba0ec874906183dca699262cda1d320"}
Oct 30 00:58:29.003253 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:29.003230    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62" event={"ID":"76501308-cde8-41f2-aa4e-b6039b294be5","Type":"ContainerStarted","Data":"5a49ae7540aad559af2607523ea7edcdae0220db820e7294392aa4de33fcdc26"}
Oct 30 00:58:29.072876 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:29.072806    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62" podStartSLOduration=2.0727848 podStartE2EDuration="2.0727848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:29.054678616 +0000 UTC m=+1021.392780527" watchObservedRunningTime="2026-10-30 00:58:29.0727848 +0000 UTC m=+1021.410886711"
Oct 30 00:58:29.127259 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:29.127220    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:29.128664 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:29.128638    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 00:58:29.128755 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:29.128696    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62" podUID="76501308-cde8-41f2-aa4e-b6039b294be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 00:58:30.127314 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:30.127283    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 00:58:30.127711 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:30.127331    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62" podUID="76501308-cde8-41f2-aa4e-b6039b294be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 00:58:31.127343 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:31.127308    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 00:58:31.127769 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:31.127378    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62" podUID="76501308-cde8-41f2-aa4e-b6039b294be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 00:58:32.127423 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:32.127383    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 00:58:32.127840 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:32.127451    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62" podUID="76501308-cde8-41f2-aa4e-b6039b294be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 00:58:33.127182 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:33.127143    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 00:58:33.127400 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:33.127201    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62" podUID="76501308-cde8-41f2-aa4e-b6039b294be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 00:58:34.126882 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:34.126843    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 00:58:34.127368 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:34.126912    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62" podUID="76501308-cde8-41f2-aa4e-b6039b294be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 00:58:35.126960 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.126928    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 00:58:35.127374 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.127005    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62" podUID="76501308-cde8-41f2-aa4e-b6039b294be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 00:58:35.541296 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.541259    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"]
Oct 30 00:58:35.545651 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.545632    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.570307 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.570274    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"]
Oct 30 00:58:35.650953 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.650913    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xtqn\" (UniqueName: \"kubernetes.io/projected/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-kube-api-access-2xtqn\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.651150 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.650973    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.651150 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.651063    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.651258 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.651163    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.651258 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.651198    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.651258 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.651221    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.651454 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.651429    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.651553 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.651536    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.651598 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.651585    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.752498 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.752459    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.752498 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.752494    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.752756 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.752536    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.752756 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.752565    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.752756 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.752586    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.752756 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.752677    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.752756 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.752744    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.753086 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.752777    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.753086 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.752808    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xtqn\" (UniqueName: \"kubernetes.io/projected/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-kube-api-access-2xtqn\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.753205 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.753136    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.753205 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.753171    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.753358 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.753334    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.753468 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.753445    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.753588 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.753477    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.755767 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.755710    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.755767 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.755735    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.774564 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.774539    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.774754 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.774732    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xtqn\" (UniqueName: \"kubernetes.io/projected/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-kube-api-access-2xtqn\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:35.875628 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:35.875547    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:36.131138 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:36.131029    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:36.131658 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:36.131449    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:36.132322 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:36.132303    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 00:58:36.189780 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:36.189750    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"]
Oct 30 00:58:36.193715 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:58:36.193691    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b4ae8be_3cac_4c96_94a8_a37a9d8cbc5f.slice/crio-db55b13497aa3ac876282c31facc676bbcd118a8f21b4f43f6bf6797bcfe30c2 WatchSource:0}: Error finding container db55b13497aa3ac876282c31facc676bbcd118a8f21b4f43f6bf6797bcfe30c2: Status 404 returned error can't find the container with id db55b13497aa3ac876282c31facc676bbcd118a8f21b4f43f6bf6797bcfe30c2
Oct 30 00:58:36.196243 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:36.196214    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:58:36.196331 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:36.196276    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:58:36.196331 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:36.196304    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:58:37.085112 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:37.084150    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z" event={"ID":"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f","Type":"ContainerStarted","Data":"fb85388065591f9af59ba4981e0a68c69819c554ec03d5df41ce90ffb0854875"}
Oct 30 00:58:37.085112 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:37.084220    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z" event={"ID":"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f","Type":"ContainerStarted","Data":"db55b13497aa3ac876282c31facc676bbcd118a8f21b4f43f6bf6797bcfe30c2"}
Oct 30 00:58:37.136679 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:37.136625    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z" podStartSLOduration=2.136611305 podStartE2EDuration="2.136611305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:37.135177257 +0000 UTC m=+1029.473279169" watchObservedRunningTime="2026-10-30 00:58:37.136611305 +0000 UTC m=+1029.474713215"
Oct 30 00:58:37.881212 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:37.881161    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:37.881994 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:37.881967    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 00:58:37.882148 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:37.882019    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z" podUID="2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 00:58:38.881216 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:38.881177    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 00:58:38.881626 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:38.881240    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z" podUID="2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 00:58:39.881545 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:39.881501    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 00:58:39.882033 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:39.881567    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z" podUID="2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 00:58:40.881494 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:40.881456    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 00:58:40.881675 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:40.881531    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z" podUID="2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 00:58:41.879095 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:41.878465    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 00:58:41.879095 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:41.878526    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z" podUID="2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 00:58:42.876322 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:42.876288    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 00:58:42.876741 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:42.876340    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z" podUID="2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 00:58:43.902423 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:43.902389    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:43.904539 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:43.904516    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:43.916633 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:43.916596    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 00:58:45.087290 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:45.087222    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh" podUID="6ba3a068-e4af-4bd1-b007-2f73f455f65c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:45.387154 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:45.383159    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29" podUID="e0ab9bcb-f48a-404b-b787-9f8c43b58722" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:45.517687 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:45.517495    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll" podUID="2388a847-21ff-46dd-8b45-4183a2448cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:48.463731 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.463694    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"]
Oct 30 00:58:48.467744 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.467723    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.496820 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.496789    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"]
Oct 30 00:58:48.637162 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.637129    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-token\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.637162 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.637165    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-data\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.637366 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.637255    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-envoy\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.637366 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.637313    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-workload-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.637437 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.637377    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-workload-certs\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.637470 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.637447    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-credential-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.637511 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.637497    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc9v5\" (UniqueName: \"kubernetes.io/projected/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-kube-api-access-gc9v5\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.637645 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.637618    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-podinfo\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.637697 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.637650    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istiod-ca-cert\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.739239 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.739151    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-envoy\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.739239 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.739217    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-workload-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.739418 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.739247    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-workload-certs\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.739418 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.739360    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-credential-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.739510 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.739423    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc9v5\" (UniqueName: \"kubernetes.io/projected/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-kube-api-access-gc9v5\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.739510 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.739489    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-podinfo\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.739625 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.739512    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istiod-ca-cert\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.739625 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.739608    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-token\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.739741 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.739642    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-data\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.739741 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.739693    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-workload-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.739741 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.739728    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-workload-certs\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.739870 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.739794    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-credential-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.740053 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.740020    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-data\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.740252 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.740231    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istiod-ca-cert\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.741584 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.741563    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-envoy\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.741947 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.741923    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-podinfo\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.761513 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.761480    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-token\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.761735 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.761720    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc9v5\" (UniqueName: \"kubernetes.io/projected/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-kube-api-access-gc9v5\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:48.782057 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:48.782034    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:49.041580 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:58:49.041505    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:58:49.098095 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:49.097920    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"]
Oct 30 00:58:49.101105 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:58:49.101063    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5461f8d_22e4_4d42_b19b_9607ddbd97a0.slice/crio-5f61d4ccd3c7364399c3bc0b49b8ca648c0ad9d032131fb4dd64f6d82d9c86b6 WatchSource:0}: Error finding container 5f61d4ccd3c7364399c3bc0b49b8ca648c0ad9d032131fb4dd64f6d82d9c86b6: Status 404 returned error can't find the container with id 5f61d4ccd3c7364399c3bc0b49b8ca648c0ad9d032131fb4dd64f6d82d9c86b6
Oct 30 00:58:49.103574 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:49.103542    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:58:49.103657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:49.103606    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:58:49.103657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:49.103641    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:58:49.271271 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:49.271230    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj" event={"ID":"e5461f8d-22e4-4d42-b19b-9607ddbd97a0","Type":"ContainerStarted","Data":"94c7a9e0f554e81d61aa4edcf3c0e3e31e0673d2c5d26421715af3faa51660c8"}
Oct 30 00:58:49.271458 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:49.271289    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj" event={"ID":"e5461f8d-22e4-4d42-b19b-9607ddbd97a0","Type":"ContainerStarted","Data":"5f61d4ccd3c7364399c3bc0b49b8ca648c0ad9d032131fb4dd64f6d82d9c86b6"}
Oct 30 00:58:49.317001 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:49.316888    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj" podStartSLOduration=1.3168730069999999 podStartE2EDuration="1.316873007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:49.313495943 +0000 UTC m=+1041.651597855" watchObservedRunningTime="2026-10-30 00:58:49.316873007 +0000 UTC m=+1041.654974894"
Oct 30 00:58:49.783188 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:49.783157    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:50.784600 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:50.784563    4864 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 00:58:50.785122 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:50.784624    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj" podUID="e5461f8d-22e4-4d42-b19b-9607ddbd97a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 00:58:51.782712 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.782675    4864 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 00:58:51.782934 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.782736    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj" podUID="e5461f8d-22e4-4d42-b19b-9607ddbd97a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 00:58:51.795124 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.795095    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"]
Oct 30 00:58:51.799202 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.799179    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.817898 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.817870    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"]
Oct 30 00:58:51.893589 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.893548    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c136a701-915b-409a-8b28-198deed55a38-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.893739 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.893603    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.893739 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.893688    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.893739 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.893731    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.893857 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.893821    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.893895 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.893877    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.893928 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.893901    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c136a701-915b-409a-8b28-198deed55a38-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.893984 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.893969    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c136a701-915b-409a-8b28-198deed55a38-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.894019 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.894002    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rbp6\" (UniqueName: \"kubernetes.io/projected/c136a701-915b-409a-8b28-198deed55a38-kube-api-access-9rbp6\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.995531 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.995499    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.995725 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.995552    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.995725 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.995587    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c136a701-915b-409a-8b28-198deed55a38-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.995725 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.995638    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c136a701-915b-409a-8b28-198deed55a38-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.995725 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.995672    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rbp6\" (UniqueName: \"kubernetes.io/projected/c136a701-915b-409a-8b28-198deed55a38-kube-api-access-9rbp6\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.995946 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.995812    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c136a701-915b-409a-8b28-198deed55a38-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.995946 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.995882    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.995946 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.995907    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.996159 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.995950    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.996159 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.995976    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.996159 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.996032    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.996321 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.996290    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.996321 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.996309    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.996596 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.996571    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c136a701-915b-409a-8b28-198deed55a38-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.998368 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.998340    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:51.998607 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:51.998589    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c136a701-915b-409a-8b28-198deed55a38-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:52.017983 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:52.017960    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c136a701-915b-409a-8b28-198deed55a38-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:52.018287 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:52.018270    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rbp6\" (UniqueName: \"kubernetes.io/projected/c136a701-915b-409a-8b28-198deed55a38-kube-api-access-9rbp6\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:52.113700 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:52.113615    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:52.454044 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:52.454015    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"]
Oct 30 00:58:52.482000 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:58:52.481957    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc136a701_915b_409a_8b28_198deed55a38.slice/crio-f097284fe643a4ceaa723ad1f14553ba87df15619733e2a2d9814896d2e0fadf WatchSource:0}: Error finding container f097284fe643a4ceaa723ad1f14553ba87df15619733e2a2d9814896d2e0fadf: Status 404 returned error can't find the container with id f097284fe643a4ceaa723ad1f14553ba87df15619733e2a2d9814896d2e0fadf
Oct 30 00:58:52.484604 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:52.484572    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:58:52.484714 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:52.484640    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:58:52.484714 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:52.484667    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:58:52.783139 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:52.783026    4864 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 00:58:52.783139 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:52.783104    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj" podUID="e5461f8d-22e4-4d42-b19b-9607ddbd97a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 00:58:53.309196 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:53.309159    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl" event={"ID":"c136a701-915b-409a-8b28-198deed55a38","Type":"ContainerStarted","Data":"4b1110e22b0a9ed98daadffa08d6094d3affbe74ba27c2923dc8913565f3bb2b"}
Oct 30 00:58:53.309653 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:53.309215    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl" event={"ID":"c136a701-915b-409a-8b28-198deed55a38","Type":"ContainerStarted","Data":"f097284fe643a4ceaa723ad1f14553ba87df15619733e2a2d9814896d2e0fadf"}
Oct 30 00:58:53.384880 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:53.384812    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl" podStartSLOduration=2.384790919 podStartE2EDuration="2.384790919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:53.363421317 +0000 UTC m=+1045.701523230" watchObservedRunningTime="2026-10-30 00:58:53.384790919 +0000 UTC m=+1045.722892836"
Oct 30 00:58:53.784529 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:53.784487    4864 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 00:58:53.784812 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:53.784769    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj" podUID="e5461f8d-22e4-4d42-b19b-9607ddbd97a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 00:58:54.113969 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:54.113881    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:58:54.115780 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:54.115251    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 00:58:54.115780 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:54.115292    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl" podUID="c136a701-915b-409a-8b28-198deed55a38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 00:58:54.784532 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:54.784490    4864 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 00:58:54.785162 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:54.785124    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj" podUID="e5461f8d-22e4-4d42-b19b-9607ddbd97a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 00:58:55.114480 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:55.114380    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 00:58:55.114480 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:55.114450    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl" podUID="c136a701-915b-409a-8b28-198deed55a38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 00:58:55.784925 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:55.784414    4864 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 00:58:55.784925 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:55.784488    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj" podUID="e5461f8d-22e4-4d42-b19b-9607ddbd97a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 00:58:56.114399 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:56.114312    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 00:58:56.114548 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:56.114398    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl" podUID="c136a701-915b-409a-8b28-198deed55a38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 00:58:56.787055 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:56.787016    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:56.787484 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:56.787409    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:56.788150 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:56.788128    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 00:58:57.114117 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:57.114024    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 00:58:57.114117 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:57.114104    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl" podUID="c136a701-915b-409a-8b28-198deed55a38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 00:58:58.114322 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:58.114288    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 00:58:58.114736 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:58.114368    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl" podUID="c136a701-915b-409a-8b28-198deed55a38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 00:58:59.114179 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:59.114131    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 00:58:59.114380 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:58:59.114193    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl" podUID="c136a701-915b-409a-8b28-198deed55a38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 00:59:00.134096 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:00.133462    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 00:59:00.134096 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:00.133524    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl" podUID="c136a701-915b-409a-8b28-198deed55a38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 00:59:00.773898 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:00.772767    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj" podUID="8b267884-baae-4ad3-a414-82467f7188b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:00.982248 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:00.982190    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj" podUID="63a8525d-9951-4f19-b124-62e4fa91c6b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:01.112214 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:01.112112    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh" podUID="6ba3a068-e4af-4bd1-b007-2f73f455f65c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Oct 30 00:59:01.518426 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:01.518393    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:59:01.520609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:01.520585    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:59:01.533424 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:01.533398    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 00:59:19.353421 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:59:19.353380    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:59:19.411096 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:19.389200    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29" podUID="e0ab9bcb-f48a-404b-b787-9f8c43b58722" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 00:59:19.525251 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:19.525193    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll" podUID="2388a847-21ff-46dd-8b45-4183a2448cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:19.633982 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:19.633873    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz" podUID="86e98e6a-b1cf-4d08-b807-26397cd1b4a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:19.654719 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:19.654678    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv" podUID="3c545315-888c-4418-8348-b06a0080a599" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:19.758245 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:19.758194    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj" podUID="8b267884-baae-4ad3-a414-82467f7188b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:19.917289 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:19.917185    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj" podUID="63a8525d-9951-4f19-b124-62e4fa91c6b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:20.134810 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:20.134681    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh" podUID="6ba3a068-e4af-4bd1-b007-2f73f455f65c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Oct 30 00:59:20.314097 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:20.277930    4864 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29" podUID="e0ab9bcb-f48a-404b-b787-9f8c43b58722" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:20.398104 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:20.395451    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29" podUID="e0ab9bcb-f48a-404b-b787-9f8c43b58722" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 00:59:20.398104 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:20.397044    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:59:20.631965 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:20.631865    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll" podUID="2388a847-21ff-46dd-8b45-4183a2448cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 30 00:59:20.834770 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:20.833003    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:59:20.912596 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:20.745464    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz" podUID="86e98e6a-b1cf-4d08-b807-26397cd1b4a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:20.953416 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:20.807220    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj" podUID="8b267884-baae-4ad3-a414-82467f7188b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 00:59:20.954578 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:20.743718    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv" podUID="3c545315-888c-4418-8348-b06a0080a599" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 00:59:20.955295 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:20.835168    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:59:20.955467 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:20.955452    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"]
Oct 30 00:59:20.988278 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:20.918165    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj" podUID="63a8525d-9951-4f19-b124-62e4fa91c6b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:21.041774 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:20.918213    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d" podUID="b460b90d-0d55-44af-a4b9-2e70b3a7b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:21.054154 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.052329    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:59:21.054154 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.052384    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:59:21.054154 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.052420    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:59:21.054154 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.052438    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"]
Oct 30 00:59:21.054154 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.052489    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:59:21.054154 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.052882    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.111097 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.104657    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:59:21.132601 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.130500    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:59:21.136661 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.136609    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:59:21.137765 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.137148    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh" podUID="6ba3a068-e4af-4bd1-b007-2f73f455f65c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:21.138700 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.138359    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:59:21.158947 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.158917    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 00:59:21.164046 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.163951    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.164046 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.164018    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.164283 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.164049    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.164283 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.164163    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db3e7716-143c-4e34-8d22-daae99a75109-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.164356 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.164317    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.164391 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.164374    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.164427 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.164405    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3e7716-143c-4e34-8d22-daae99a75109-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.164506 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.164488    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db3e7716-143c-4e34-8d22-daae99a75109-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.164541 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.164527    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvwpl\" (UniqueName: \"kubernetes.io/projected/db3e7716-143c-4e34-8d22-daae99a75109-kube-api-access-rvwpl\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.168885 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.168866    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:59:21.171638 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.171620    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:59:21.177110 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.174771    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:59:21.265345 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.265316    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db3e7716-143c-4e34-8d22-daae99a75109-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.265532 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.265413    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.265532 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.265438    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.265532 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.265460    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3e7716-143c-4e34-8d22-daae99a75109-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.265532 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.265526    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db3e7716-143c-4e34-8d22-daae99a75109-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.265761 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.265549    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvwpl\" (UniqueName: \"kubernetes.io/projected/db3e7716-143c-4e34-8d22-daae99a75109-kube-api-access-rvwpl\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.265761 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.265670    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.265761 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.265720    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.265761 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.265745    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.266050 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.265900    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.266130 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.266067    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.266188 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.266134    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.266253 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.266191    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.266253 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.266238    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db3e7716-143c-4e34-8d22-daae99a75109-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.267989 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.267966    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.268173 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.268156    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db3e7716-143c-4e34-8d22-daae99a75109-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.292412 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.292386    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvwpl\" (UniqueName: \"kubernetes.io/projected/db3e7716-143c-4e34-8d22-daae99a75109-kube-api-access-rvwpl\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.292546 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.292443    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3e7716-143c-4e34-8d22-daae99a75109-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.423232 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.423153    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:21.744866 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.744835    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"]
Oct 30 00:59:21.748627 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:59:21.748601    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb3e7716_143c_4e34_8d22_daae99a75109.slice/crio-301123f0fab905562bc6f817eaca645887fa4f77fee227ab08813ff7729c0650 WatchSource:0}: Error finding container 301123f0fab905562bc6f817eaca645887fa4f77fee227ab08813ff7729c0650: Status 404 returned error can't find the container with id 301123f0fab905562bc6f817eaca645887fa4f77fee227ab08813ff7729c0650
Oct 30 00:59:21.751219 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.751185    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:59:21.751318 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.751256    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:59:21.751318 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:21.751285    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:59:22.072466 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:22.072365    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7" event={"ID":"db3e7716-143c-4e34-8d22-daae99a75109","Type":"ContainerStarted","Data":"61b2c1602aafc4148bb8e91e1e5c61a39565d65348270b039f3a742e168835b6"}
Oct 30 00:59:22.072466 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:22.072429    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7" event={"ID":"db3e7716-143c-4e34-8d22-daae99a75109","Type":"ContainerStarted","Data":"301123f0fab905562bc6f817eaca645887fa4f77fee227ab08813ff7729c0650"}
Oct 30 00:59:22.119278 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:22.119217    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7" podStartSLOduration=2.119201592 podStartE2EDuration="2.119201592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:22.113162131 +0000 UTC m=+1074.451264043" watchObservedRunningTime="2026-10-30 00:59:22.119201592 +0000 UTC m=+1074.457303502"
Oct 30 00:59:22.423389 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:22.423305    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:22.424883 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:22.424854    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:59:22.425026 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:22.424910    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7" podUID="db3e7716-143c-4e34-8d22-daae99a75109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 00:59:23.423911 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:23.423873    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:59:23.424342 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:23.423929    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7" podUID="db3e7716-143c-4e34-8d22-daae99a75109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 00:59:24.424429 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:24.424391    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:59:24.424804 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:24.424446    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7" podUID="db3e7716-143c-4e34-8d22-daae99a75109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 00:59:25.424255 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:25.424220    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:59:25.424440 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:25.424274    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7" podUID="db3e7716-143c-4e34-8d22-daae99a75109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 00:59:26.048349 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.048317    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"]
Oct 30 00:59:26.053862 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.053843    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.085699 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.085667    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"]
Oct 30 00:59:26.140676 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.140645    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.140676 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.140682    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.140894 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.140706    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.140894 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.140820    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.140968 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.140911    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.140968 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.140949    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.141131 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.141091    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.141131 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.141127    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dfc55de-0824-44de-a268-b2fcd2ea193f-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.141218 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.141171    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p897n\" (UniqueName: \"kubernetes.io/projected/6dfc55de-0824-44de-a268-b2fcd2ea193f-kube-api-access-p897n\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.242894 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.242859    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.243112 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.242955    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.243112 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.242979    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.243112 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.243060    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.243366 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.243190    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dfc55de-0824-44de-a268-b2fcd2ea193f-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.243366 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.243232    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p897n\" (UniqueName: \"kubernetes.io/projected/6dfc55de-0824-44de-a268-b2fcd2ea193f-kube-api-access-p897n\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.243366 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.243336    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.243521 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.243368    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.243521 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.243385    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.243521 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.243408    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.243521 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.243394    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.243521 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.243453    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.243521 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.243485    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.244180 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.244154    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dfc55de-0824-44de-a268-b2fcd2ea193f-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.245511 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.245494    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.245929 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.245906    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.285017 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.284986    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p897n\" (UniqueName: \"kubernetes.io/projected/6dfc55de-0824-44de-a268-b2fcd2ea193f-kube-api-access-p897n\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.285427 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.285375    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.401152 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.401028    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:26.423611 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.423582    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:59:26.423742 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.423641    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7" podUID="db3e7716-143c-4e34-8d22-daae99a75109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 00:59:26.726847 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.726810    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"]
Oct 30 00:59:26.729967 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:59:26.729938    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dfc55de_0824_44de_a268_b2fcd2ea193f.slice/crio-3266ba304dbfe49dda50485ce46213436709ac128cfb6caa233e4a465ff0c125 WatchSource:0}: Error finding container 3266ba304dbfe49dda50485ce46213436709ac128cfb6caa233e4a465ff0c125: Status 404 returned error can't find the container with id 3266ba304dbfe49dda50485ce46213436709ac128cfb6caa233e4a465ff0c125
Oct 30 00:59:26.732557 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.732519    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:59:26.732653 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.732590    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:59:26.732653 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:26.732621    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:59:27.116218 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:27.116108    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778" event={"ID":"6dfc55de-0824-44de-a268-b2fcd2ea193f","Type":"ContainerStarted","Data":"7c2b2854b2d9fb031078ad57c308b5d530506a660f7ca74b503d2a944c659ae5"}
Oct 30 00:59:27.116600 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:27.116221    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778" event={"ID":"6dfc55de-0824-44de-a268-b2fcd2ea193f","Type":"ContainerStarted","Data":"3266ba304dbfe49dda50485ce46213436709ac128cfb6caa233e4a465ff0c125"}
Oct 30 00:59:27.190026 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:27.189972    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778" podStartSLOduration=1.189955691 podStartE2EDuration="1.189955691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:27.177508635 +0000 UTC m=+1079.515610546" watchObservedRunningTime="2026-10-30 00:59:27.189955691 +0000 UTC m=+1079.528057602"
Oct 30 00:59:27.402207 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:27.402117    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:27.403611 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:27.403578    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:59:27.403775 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:27.403641    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778" podUID="6dfc55de-0824-44de-a268-b2fcd2ea193f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 00:59:27.424247 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:27.424201    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:59:27.424380 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:27.424284    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7" podUID="db3e7716-143c-4e34-8d22-daae99a75109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 00:59:28.401525 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:28.401472    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:59:28.402021 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:28.401564    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778" podUID="6dfc55de-0824-44de-a268-b2fcd2ea193f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 00:59:28.424502 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:28.424470    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:59:28.424637 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:28.424524    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7" podUID="db3e7716-143c-4e34-8d22-daae99a75109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 00:59:29.402238 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:29.402179    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:59:29.402660 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:29.402257    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778" podUID="6dfc55de-0824-44de-a268-b2fcd2ea193f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 00:59:29.428852 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:29.428817    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:29.429323 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:29.429309    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:29.430053 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:29.430036    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 00:59:30.401798 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:30.401761    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:59:30.401988 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:30.401817    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778" podUID="6dfc55de-0824-44de-a268-b2fcd2ea193f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 00:59:31.403097 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:31.401456    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:59:31.403097 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:31.401522    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778" podUID="6dfc55de-0824-44de-a268-b2fcd2ea193f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 00:59:32.418554 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:32.418513    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:59:32.419590 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:32.419543    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778" podUID="6dfc55de-0824-44de-a268-b2fcd2ea193f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 00:59:33.132099 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:33.117346    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj" podUID="63a8525d-9951-4f19-b124-62e4fa91c6b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:33.133138 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:33.133096    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz" podUID="86e98e6a-b1cf-4d08-b807-26397cd1b4a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:33.142145 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:33.137264    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll" podUID="2388a847-21ff-46dd-8b45-4183a2448cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 30 00:59:33.180636 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:33.176159    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj" podUID="8b267884-baae-4ad3-a414-82467f7188b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 00:59:33.180636 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:33.177496    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29" podUID="e0ab9bcb-f48a-404b-b787-9f8c43b58722" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:33.180636 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:33.177777    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh" podUID="6ba3a068-e4af-4bd1-b007-2f73f455f65c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Oct 30 00:59:33.192241 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:33.192200    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv" podUID="3c545315-888c-4418-8348-b06a0080a599" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:33.405584 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:33.405486    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:59:33.482103 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:33.474697    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778" podUID="6dfc55de-0824-44de-a268-b2fcd2ea193f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 00:59:34.163458 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:34.163397    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj" podUID="63a8525d-9951-4f19-b124-62e4fa91c6b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:34.164516 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:34.164494    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:59:34.174501 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:34.174461    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll" podUID="2388a847-21ff-46dd-8b45-4183a2448cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 30 00:59:34.183232 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:34.174986    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz" podUID="86e98e6a-b1cf-4d08-b807-26397cd1b4a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 30 00:59:34.206098 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:34.176666    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:59:34.443154 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:34.410202    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:59:34.476538 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:34.187166    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh" podUID="6ba3a068-e4af-4bd1-b007-2f73f455f65c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Oct 30 00:59:34.537160 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:34.537130    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:59:34.596299 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:34.177711    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29" podUID="e0ab9bcb-f48a-404b-b787-9f8c43b58722" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:34.597470 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:34.435320    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:59:34.597669 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:34.597644    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778" podUID="6dfc55de-0824-44de-a268-b2fcd2ea193f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 00:59:34.597760 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:34.209165    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv" podUID="3c545315-888c-4418-8348-b06a0080a599" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:34.600095 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:34.598990    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:59:34.605193 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:34.605169    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:59:34.743149 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:34.727889    4864 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll" podUID="2388a847-21ff-46dd-8b45-4183a2448cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:35.199409 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:35.167689    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj" podUID="63a8525d-9951-4f19-b124-62e4fa91c6b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:35.199409 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:35.180318    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll" podUID="2388a847-21ff-46dd-8b45-4183a2448cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 30 00:59:35.219435 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:35.219392    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz" podUID="86e98e6a-b1cf-4d08-b807-26397cd1b4a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 30 00:59:35.339881 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:35.339834    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj" podUID="8b267884-baae-4ad3-a414-82467f7188b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:35.427656 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:35.427613    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:59:35.427858 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:35.427681    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778" podUID="6dfc55de-0824-44de-a268-b2fcd2ea193f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 00:59:35.536216 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:35.536168    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh" podUID="6ba3a068-e4af-4bd1-b007-2f73f455f65c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:35.600462 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:35.600401    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv" podUID="3c545315-888c-4418-8348-b06a0080a599" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:35.604724 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:35.604685    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29" podUID="e0ab9bcb-f48a-404b-b787-9f8c43b58722" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 00:59:35.650620 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:35.650582    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 00:59:35.652015 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:35.651977    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 00:59:35.652741 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:35.652721    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 00:59:35.653091 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:35.653063    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 00:59:35.655171 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:35.655139    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 00:59:35.655341 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:35.655321    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 00:59:36.402279 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:36.402241    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:59:36.402462 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:36.402307    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778" podUID="6dfc55de-0824-44de-a268-b2fcd2ea193f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 00:59:37.406188 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:37.406160    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:37.406603 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:37.406559    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:37.407305 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:37.407286    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 00:59:41.761720 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:41.761687    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9"]
Oct 30 00:59:41.767013 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:41.766989    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9"
Oct 30 00:59:41.780424 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:41.780395    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8363a40b-serving-cert\""
Oct 30 00:59:41.800346 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:41.800311    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9"]
Oct 30 00:59:41.907919 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:41.907878    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b21e6da-169c-494d-8afa-cb8288ebdeb0-openshift-service-ca\") pod \"maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9\" (UID: \"8b21e6da-169c-494d-8afa-cb8288ebdeb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9"
Oct 30 00:59:41.908129 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:41.908038    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5gc7\" (UniqueName: \"kubernetes.io/projected/8b21e6da-169c-494d-8afa-cb8288ebdeb0-kube-api-access-x5gc7\") pod \"maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9\" (UID: \"8b21e6da-169c-494d-8afa-cb8288ebdeb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9"
Oct 30 00:59:41.908229 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:41.908210    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b21e6da-169c-494d-8afa-cb8288ebdeb0-metrics-certs\") pod \"maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9\" (UID: \"8b21e6da-169c-494d-8afa-cb8288ebdeb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9"
Oct 30 00:59:41.908701 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:41.908678    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b21e6da-169c-494d-8afa-cb8288ebdeb0-maas-api-tls\") pod \"maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9\" (UID: \"8b21e6da-169c-494d-8afa-cb8288ebdeb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9"
Oct 30 00:59:42.014152 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:42.014055    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b21e6da-169c-494d-8afa-cb8288ebdeb0-maas-api-tls\") pod \"maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9\" (UID: \"8b21e6da-169c-494d-8afa-cb8288ebdeb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9"
Oct 30 00:59:42.014347 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:42.014154    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b21e6da-169c-494d-8afa-cb8288ebdeb0-openshift-service-ca\") pod \"maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9\" (UID: \"8b21e6da-169c-494d-8afa-cb8288ebdeb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9"
Oct 30 00:59:42.014347 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:42.014327    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5gc7\" (UniqueName: \"kubernetes.io/projected/8b21e6da-169c-494d-8afa-cb8288ebdeb0-kube-api-access-x5gc7\") pod \"maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9\" (UID: \"8b21e6da-169c-494d-8afa-cb8288ebdeb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9"
Oct 30 00:59:42.014475 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:42.014440    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b21e6da-169c-494d-8afa-cb8288ebdeb0-metrics-certs\") pod \"maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9\" (UID: \"8b21e6da-169c-494d-8afa-cb8288ebdeb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9"
Oct 30 00:59:42.014985 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:42.014946    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b21e6da-169c-494d-8afa-cb8288ebdeb0-openshift-service-ca\") pod \"maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9\" (UID: \"8b21e6da-169c-494d-8afa-cb8288ebdeb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9"
Oct 30 00:59:42.017608 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:42.017578    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b21e6da-169c-494d-8afa-cb8288ebdeb0-metrics-certs\") pod \"maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9\" (UID: \"8b21e6da-169c-494d-8afa-cb8288ebdeb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9"
Oct 30 00:59:42.017707 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:42.017634    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b21e6da-169c-494d-8afa-cb8288ebdeb0-maas-api-tls\") pod \"maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9\" (UID: \"8b21e6da-169c-494d-8afa-cb8288ebdeb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9"
Oct 30 00:59:42.036557 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:42.036521    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5gc7\" (UniqueName: \"kubernetes.io/projected/8b21e6da-169c-494d-8afa-cb8288ebdeb0-kube-api-access-x5gc7\") pod \"maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9\" (UID: \"8b21e6da-169c-494d-8afa-cb8288ebdeb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9"
Oct 30 00:59:42.080687 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:42.080659    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9"
Oct 30 00:59:42.386190 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:42.386112    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9"]
Oct 30 00:59:42.389952 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:59:42.389916    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b21e6da_169c_494d_8afa_cb8288ebdeb0.slice/crio-ceb9326bbfe487c4b2b57aed85ad7ebd10944245b25f5ed69a5a534feba8a3d4 WatchSource:0}: Error finding container ceb9326bbfe487c4b2b57aed85ad7ebd10944245b25f5ed69a5a534feba8a3d4: Status 404 returned error can't find the container with id ceb9326bbfe487c4b2b57aed85ad7ebd10944245b25f5ed69a5a534feba8a3d4
Oct 30 00:59:42.391395 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:42.391376    4864 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:59:42.587832 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:42.587793    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9" event={"ID":"8b21e6da-169c-494d-8afa-cb8288ebdeb0","Type":"ContainerStarted","Data":"ceb9326bbfe487c4b2b57aed85ad7ebd10944245b25f5ed69a5a534feba8a3d4"}
Oct 30 00:59:43.594996 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:43.594952    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9" event={"ID":"8b21e6da-169c-494d-8afa-cb8288ebdeb0","Type":"ContainerStarted","Data":"f0b8cb847d076248d7fed0621a90105531c1812d63b4f1eac734b359cbde97d8"}
Oct 30 00:59:43.595423 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:43.595053    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9"
Oct 30 00:59:43.685633 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:43.685581    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9" podStartSLOduration=2.26944276 podStartE2EDuration="2.685566998s" totalImagesPullingTime="416.124238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:43.684166779 +0000 UTC m=+1096.022268688" watchObservedRunningTime="2026-10-30 00:59:43.685566998 +0000 UTC m=+1096.023668908"
Oct 30 00:59:43.943561 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:43.943479    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"]
Oct 30 00:59:43.943710 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:43.943690    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79" podUID="8fac8a72-32dd-4801-94bf-65e06ef2f8ff" containerName="istio-proxy" containerID="cri-o://1fbd64ce194902e1692dd89adc274e2b01a42ccefaf858f40b406501d71a72aa" gracePeriod=30
Oct 30 00:59:46.168265 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:46.168207    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj" podUID="8b267884-baae-4ad3-a414-82467f7188b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 00:59:46.189400 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:46.189358    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll" podUID="2388a847-21ff-46dd-8b45-4183a2448cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 30 00:59:46.230221 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:46.230160    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz" podUID="86e98e6a-b1cf-4d08-b807-26397cd1b4a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:49.041157 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:59:49.041117    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 00:59:49.222082 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.222006    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:59:49.281472 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.281436    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-credential-socket\") pod \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") "
Oct 30 00:59:49.281657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.281495    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-data\") pod \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") "
Oct 30 00:59:49.281657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.281542    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-workload-socket\") pod \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") "
Oct 30 00:59:49.281657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.281570    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-podinfo\") pod \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") "
Oct 30 00:59:49.281657 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.281642    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istiod-ca-cert\") pod \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") "
Oct 30 00:59:49.281854 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.281673    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-workload-certs\") pod \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") "
Oct 30 00:59:49.281854 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.281733    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-kube-api-access-qxs45\" (UniqueName: \"kubernetes.io/projected/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-kube-api-access-qxs45\") pod \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") "
Oct 30 00:59:49.281854 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.281841    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-token\" (UniqueName: \"kubernetes.io/projected/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-token\") pod \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") "
Oct 30 00:59:49.281993 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.281884    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-envoy\") pod \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\" (UID: \"8fac8a72-32dd-4801-94bf-65e06ef2f8ff\") "
Oct 30 00:59:49.282428 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.282400    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-data" pod "8fac8a72-32dd-4801-94bf-65e06ef2f8ff" (UID: "8fac8a72-32dd-4801-94bf-65e06ef2f8ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:49.282766 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.282395    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-workload-socket" pod "8fac8a72-32dd-4801-94bf-65e06ef2f8ff" (UID: "8fac8a72-32dd-4801-94bf-65e06ef2f8ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:49.282766 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.282605    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-workload-certs" pod "8fac8a72-32dd-4801-94bf-65e06ef2f8ff" (UID: "8fac8a72-32dd-4801-94bf-65e06ef2f8ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:49.282982 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.282915    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istiod-ca-cert" pod "8fac8a72-32dd-4801-94bf-65e06ef2f8ff" (UID: "8fac8a72-32dd-4801-94bf-65e06ef2f8ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:49.282982 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.282954    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-credential-socket" pod "8fac8a72-32dd-4801-94bf-65e06ef2f8ff" (UID: "8fac8a72-32dd-4801-94bf-65e06ef2f8ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:49.287135 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.287097    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-podinfo" pod "8fac8a72-32dd-4801-94bf-65e06ef2f8ff" (UID: "8fac8a72-32dd-4801-94bf-65e06ef2f8ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:59:49.287493 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.287459    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-token" pod "8fac8a72-32dd-4801-94bf-65e06ef2f8ff" (UID: "8fac8a72-32dd-4801-94bf-65e06ef2f8ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:49.287748 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.287713    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-kube-api-access-qxs45" pod "8fac8a72-32dd-4801-94bf-65e06ef2f8ff" (UID: "8fac8a72-32dd-4801-94bf-65e06ef2f8ff"). InnerVolumeSpecName "kube-api-access-qxs45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:49.287849 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.287825    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-envoy" pod "8fac8a72-32dd-4801-94bf-65e06ef2f8ff" (UID: "8fac8a72-32dd-4801-94bf-65e06ef2f8ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:49.383850 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.383812    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:59:49.383850 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.383844    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:59:49.383850 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.383853    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:59:49.384143 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.383862    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:59:49.384143 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.383870    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:59:49.384143 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.383880    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:59:49.384143 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.383888    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:59:49.384143 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.383897    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:59:49.384143 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.383905    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxs45\" (UniqueName: \"kubernetes.io/projected/8fac8a72-32dd-4801-94bf-65e06ef2f8ff-kube-api-access-qxs45\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 00:59:49.620948 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.620879    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9"
Oct 30 00:59:49.730029 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.729986    4864 generic.go:356] "Generic (PLEG): container finished" podID="8fac8a72-32dd-4801-94bf-65e06ef2f8ff" containerID="1fbd64ce194902e1692dd89adc274e2b01a42ccefaf858f40b406501d71a72aa" exitCode=0
Oct 30 00:59:49.730246 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.730034    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"
Oct 30 00:59:49.730246 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.730171    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79" event={"ID":"8fac8a72-32dd-4801-94bf-65e06ef2f8ff","Type":"ContainerDied","Data":"1fbd64ce194902e1692dd89adc274e2b01a42ccefaf858f40b406501d71a72aa"}
Oct 30 00:59:49.730246 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.730242    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79" event={"ID":"8fac8a72-32dd-4801-94bf-65e06ef2f8ff","Type":"ContainerDied","Data":"b74fecf7bce1e041bfd15a74f0f8e73fe43b68dda72304c53b737c81b81a373e"}
Oct 30 00:59:49.730432 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.730278    4864 scope.go:122] "RemoveContainer" containerID="1fbd64ce194902e1692dd89adc274e2b01a42ccefaf858f40b406501d71a72aa"
Oct 30 00:59:49.743986 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.743962    4864 scope.go:122] "RemoveContainer" containerID="1fbd64ce194902e1692dd89adc274e2b01a42ccefaf858f40b406501d71a72aa"
Oct 30 00:59:49.744328 ip-10-0-134-210 kubenswrapper[4864]: E1030 00:59:49.744304    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fbd64ce194902e1692dd89adc274e2b01a42ccefaf858f40b406501d71a72aa\": container with ID starting with 1fbd64ce194902e1692dd89adc274e2b01a42ccefaf858f40b406501d71a72aa not found: ID does not exist" containerID="1fbd64ce194902e1692dd89adc274e2b01a42ccefaf858f40b406501d71a72aa"
Oct 30 00:59:49.744398 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.744340    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fbd64ce194902e1692dd89adc274e2b01a42ccefaf858f40b406501d71a72aa"} err="failed to get container status \"1fbd64ce194902e1692dd89adc274e2b01a42ccefaf858f40b406501d71a72aa\": rpc error: code = NotFound desc = could not find container \"1fbd64ce194902e1692dd89adc274e2b01a42ccefaf858f40b406501d71a72aa\": container with ID starting with 1fbd64ce194902e1692dd89adc274e2b01a42ccefaf858f40b406501d71a72aa not found: ID does not exist"
Oct 30 00:59:49.779806 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.779774    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"]
Oct 30 00:59:49.784049 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.784021    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b6c79"]
Oct 30 00:59:49.793609 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:49.793578    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fac8a72-32dd-4801-94bf-65e06ef2f8ff" path="/var/lib/kubelet/pods/8fac8a72-32dd-4801-94bf-65e06ef2f8ff/volumes"
Oct 30 00:59:50.062662 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.062624    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"]
Oct 30 00:59:50.063534 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.063512    4864 cpu_manager.go:414] "containerMap: removing container" podUID="8fac8a72-32dd-4801-94bf-65e06ef2f8ff" containerName="istio-proxy"
Oct 30 00:59:50.063711 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.063689    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fac8a72-32dd-4801-94bf-65e06ef2f8ff" containerName="istio-proxy"
Oct 30 00:59:50.063902 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.063888    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fac8a72-32dd-4801-94bf-65e06ef2f8ff" containerName="istio-proxy"
Oct 30 00:59:50.068995 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.068971    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.088303 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.088275    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"]
Oct 30 00:59:50.092846 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.092808    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-token\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.092978 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.092958    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-podinfo\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.093140 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.093114    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-workload-certs\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.093235 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.093153    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-data\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.093235 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.093220    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-credential-socket\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.093347 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.093272    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-envoy\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.093347 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.093290    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frxnh\" (UniqueName: \"kubernetes.io/projected/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-kube-api-access-frxnh\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.093347 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.093318    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istiod-ca-cert\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.093499 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.093424    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-workload-socket\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.145576 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.145545    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-52b947-openshift-default-dockercfg-xcnf6\""
Oct 30 00:59:50.194360 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.194330    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-podinfo\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.194579 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.194414    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-workload-certs\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.194579 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.194438    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-data\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.194579 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.194478    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-credential-socket\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.194579 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.194537    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-envoy\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.194579 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.194558    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frxnh\" (UniqueName: \"kubernetes.io/projected/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-kube-api-access-frxnh\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.194833 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.194592    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istiod-ca-cert\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.194833 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.194678    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-workload-socket\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.194833 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.194748    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-token\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.195484 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.195453    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-credential-socket\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.195931 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.195907    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-data\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.196065 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.196004    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-workload-certs\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.196065 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.196040    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istiod-ca-cert\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.196304 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.196285    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-workload-socket\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.198154 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.198113    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-podinfo\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.198320 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.198164    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-envoy\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.216463 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.216440    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-token\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.216582 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.216516    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frxnh\" (UniqueName: \"kubernetes.io/projected/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-kube-api-access-frxnh\") pod \"e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") " pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.385081 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.384985    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:50.702102 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.702057    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"]
Oct 30 00:59:50.706059 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:59:50.706034    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eedbfd0_0fc7_43dc_a709_6a1dd6acd1ab.slice/crio-7dca8274f4c8c1ec7a966ff02e2cd894b48e004ffc1895da729cb0b4f4565eed WatchSource:0}: Error finding container 7dca8274f4c8c1ec7a966ff02e2cd894b48e004ffc1895da729cb0b4f4565eed: Status 404 returned error can't find the container with id 7dca8274f4c8c1ec7a966ff02e2cd894b48e004ffc1895da729cb0b4f4565eed
Oct 30 00:59:50.708536 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.708503    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:59:50.708605 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.708577    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:59:50.708650 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.708611    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:59:50.738439 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:50.738406    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp" event={"ID":"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab","Type":"ContainerStarted","Data":"7dca8274f4c8c1ec7a966ff02e2cd894b48e004ffc1895da729cb0b4f4565eed"}
Oct 30 00:59:51.752240 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:51.752204    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp" event={"ID":"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab","Type":"ContainerStarted","Data":"70f8fbcb166f228c99e51626618b0fead6c49eb15873f416e2daca3206da0721"}
Oct 30 00:59:51.805346 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:51.805296    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp" podStartSLOduration=1.805281919 podStartE2EDuration="1.805281919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:51.804459893 +0000 UTC m=+1104.142561803" watchObservedRunningTime="2026-10-30 00:59:51.805281919 +0000 UTC m=+1104.143383826"
Oct 30 00:59:51.969278 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:51.969244    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"]
Oct 30 00:59:51.973537 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:51.973516    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:51.995694 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:51.995659    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"]
Oct 30 00:59:52.019674 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.019416    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.019674 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.019533    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.019674 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.019571    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f83fb2-248a-49c8-ba87-ee314ff0006e-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.019674 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.019607    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbt7b\" (UniqueName: \"kubernetes.io/projected/67f83fb2-248a-49c8-ba87-ee314ff0006e-kube-api-access-dbt7b\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.019674 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.019652    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.020063 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.019848    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.020063 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.019877    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.020063 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.020038    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.020342 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.020112    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.121648 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.121613    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbt7b\" (UniqueName: \"kubernetes.io/projected/67f83fb2-248a-49c8-ba87-ee314ff0006e-kube-api-access-dbt7b\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.121848 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.121670    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.121848 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.121711    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.121848 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.121731    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.121848 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.121807    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.121848 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.121840    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.122153 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.121938    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.122153 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.122041    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.122153 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.122094    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f83fb2-248a-49c8-ba87-ee314ff0006e-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.122334 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.122249    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.122334 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.122292    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.122409 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.122340    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.122409 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.122365    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.122755 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.122727    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f83fb2-248a-49c8-ba87-ee314ff0006e-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.124320 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.124293    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.124533 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.124514    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.143907 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.143880    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.144065 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.144027    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbt7b\" (UniqueName: \"kubernetes.io/projected/67f83fb2-248a-49c8-ba87-ee314ff0006e-kube-api-access-dbt7b\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.291834 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.291761    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:52.385640 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.385609    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:52.390831 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.390806    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:52.607335 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.607108    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"]
Oct 30 00:59:52.610313 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:59:52.610276    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f83fb2_248a_49c8_ba87_ee314ff0006e.slice/crio-634e86b3b47092eeb9a7e06723c3907bddcbb95994118c77dbb590405612475f WatchSource:0}: Error finding container 634e86b3b47092eeb9a7e06723c3907bddcbb95994118c77dbb590405612475f: Status 404 returned error can't find the container with id 634e86b3b47092eeb9a7e06723c3907bddcbb95994118c77dbb590405612475f
Oct 30 00:59:52.612934 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.612902    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:59:52.613044 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.612980    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:59:52.613044 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.613024    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:59:52.761180 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.761134    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n" event={"ID":"67f83fb2-248a-49c8-ba87-ee314ff0006e","Type":"ContainerStarted","Data":"2d930fd4af2e9d74b91b410e8662e7fa006307b800e8034c7a50e0ff90adf6df"}
Oct 30 00:59:52.761602 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.761196    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n" event={"ID":"67f83fb2-248a-49c8-ba87-ee314ff0006e","Type":"ContainerStarted","Data":"634e86b3b47092eeb9a7e06723c3907bddcbb95994118c77dbb590405612475f"}
Oct 30 00:59:52.761666 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.761628    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:52.762576 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.762555    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 00:59:52.802217 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:52.802158    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n" podStartSLOduration=1.8021445809999999 podStartE2EDuration="1.802144581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:52.801544289 +0000 UTC m=+1105.139646263" watchObservedRunningTime="2026-10-30 00:59:52.802144581 +0000 UTC m=+1105.140246490"
Oct 30 00:59:53.154259 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.154227    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"]
Oct 30 00:59:53.175092 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.174363    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:53.192702 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.187483    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"]
Oct 30 00:59:53.192702 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.191791    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 00:59:53.192702 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.191855    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 00:59:53.192702 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.191960    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-htf9g\""
Oct 30 00:59:53.192702 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.192005    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 00:59:53.239228 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.239198    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgrq5\" (UniqueName: \"kubernetes.io/projected/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-kube-api-access-wgrq5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6xv9\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:53.239228 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.239232    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6xv9\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:53.239428 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.239390    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6xv9\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:53.239428 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.239414    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6xv9\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:53.239528 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.239429    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6xv9\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:53.239590 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.239567    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6xv9\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:53.239698 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.239680    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6xv9\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:53.292773 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.292743    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 00:59:53.294161 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.294142    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:59:53.294244 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.294184    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n" podUID="67f83fb2-248a-49c8-ba87-ee314ff0006e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 00:59:53.341300 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.341265    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6xv9\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:53.341450 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.341353    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6xv9\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:53.341450 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.341415    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgrq5\" (UniqueName: \"kubernetes.io/projected/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-kube-api-access-wgrq5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6xv9\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:53.341572 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.341449    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6xv9\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:53.341630 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.341593    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6xv9\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:53.341685 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.341631    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6xv9\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:53.341685 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.341656    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6xv9\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:53.342462 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.342429    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6xv9\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:53.343782 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.343753    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6xv9\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:53.344342 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.344318    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6xv9\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:53.346612 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.344857    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6xv9\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:53.346612 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.345700    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6xv9\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:53.384167 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.384127    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6xv9\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:53.385334 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.385302    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgrq5\" (UniqueName: \"kubernetes.io/projected/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-kube-api-access-wgrq5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6xv9\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:53.502577 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.502537    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:53.883346 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.883317    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"]
Oct 30 00:59:53.886354 ip-10-0-134-210 kubenswrapper[4864]: W1030 00:59:53.886317    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d4a7e1_5c2b_41a4_b2e0_73a3048ec79f.slice/crio-e8d731782a217c9576a1c090cbbb6ef18a36eb8ce28f41dff39d04eb34dba8e4 WatchSource:0}: Error finding container e8d731782a217c9576a1c090cbbb6ef18a36eb8ce28f41dff39d04eb34dba8e4: Status 404 returned error can't find the container with id e8d731782a217c9576a1c090cbbb6ef18a36eb8ce28f41dff39d04eb34dba8e4
Oct 30 00:59:53.889468 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.889429    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:59:53.889546 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:53.889509    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 00:59:54.292617 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:54.292581    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:59:54.292770 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:54.292645    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n" podUID="67f83fb2-248a-49c8-ba87-ee314ff0006e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 00:59:54.796099 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:54.796041    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9" event={"ID":"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f","Type":"ContainerStarted","Data":"4ea8d039f683a3598bb38cf1c942b68a4872dfb5db511d6c9490343d07c5f2fd"}
Oct 30 00:59:54.796280 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:54.796114    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9" event={"ID":"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f","Type":"ContainerStarted","Data":"e8d731782a217c9576a1c090cbbb6ef18a36eb8ce28f41dff39d04eb34dba8e4"}
Oct 30 00:59:54.796280 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:54.796258    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:54.798360 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:54.798328    4864 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-p6xv9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 00:59:54.798507 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:54.798386    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9" podUID="63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 00:59:54.872709 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:54.872637    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9" podStartSLOduration=1.87261489 podStartE2EDuration="1.87261489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:54.84777746 +0000 UTC m=+1107.185879371" watchObservedRunningTime="2026-10-30 00:59:54.87261489 +0000 UTC m=+1107.210716801"
Oct 30 00:59:55.292586 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:55.292547    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:59:55.293109 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:55.292612    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n" podUID="67f83fb2-248a-49c8-ba87-ee314ff0006e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 00:59:55.802286 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:55.802261    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 00:59:56.310912 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:56.310876    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:59:56.311586 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:56.311551    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n" podUID="67f83fb2-248a-49c8-ba87-ee314ff0006e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 00:59:57.292601 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:57.292566    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:59:57.292777 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:57.292637    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n" podUID="67f83fb2-248a-49c8-ba87-ee314ff0006e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 00:59:58.292353 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:58.292315    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:59:58.292753 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:58.292389    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n" podUID="67f83fb2-248a-49c8-ba87-ee314ff0006e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 00:59:59.293006 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:59.292967    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:59:59.293563 ip-10-0-134-210 kubenswrapper[4864]: I1030 00:59:59.293047    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n" podUID="67f83fb2-248a-49c8-ba87-ee314ff0006e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 01:00:00.296824 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:00.296797    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 01:00:00.297329 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:00.297312    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 01:00:00.298026 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:00.298005    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 01:00:14.044232 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.044150    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"]
Oct 30 01:00:14.049658 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.049635    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.062878 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.062855    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-spn7j\""
Oct 30 01:00:14.077663 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.077630    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"]
Oct 30 01:00:14.164032 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.163995    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.164252 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.164064    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99s4b\" (UniqueName: \"kubernetes.io/projected/b5bec735-88e4-4b14-899d-5e8158d2638c-kube-api-access-99s4b\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.164252 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.164161    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.164371 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.164270    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.164371 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.164292    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.164371 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.164338    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.164539 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.164418    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5bec735-88e4-4b14-899d-5e8158d2638c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.164539 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.164516    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.164638 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.164541    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.265419 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.265377    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.265419 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.265424    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.265680 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.265480    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.265680 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.265542    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5bec735-88e4-4b14-899d-5e8158d2638c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.265680 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.265594    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.265680 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.265615    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.265888 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.265706    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.265888 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.265783    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99s4b\" (UniqueName: \"kubernetes.io/projected/b5bec735-88e4-4b14-899d-5e8158d2638c-kube-api-access-99s4b\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.265888 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.265820    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.266038 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.265943    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.266038 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.266017    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.266162 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.266135    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.266217 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.266174    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.266333 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.266315    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5bec735-88e4-4b14-899d-5e8158d2638c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.267997 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.267973    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.268368 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.268348    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.288840 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.288814    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99s4b\" (UniqueName: \"kubernetes.io/projected/b5bec735-88e4-4b14-899d-5e8158d2638c-kube-api-access-99s4b\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.288954 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.288856    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8s5bq\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.365222 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.365135    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:14.687275 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.687248    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"]
Oct 30 01:00:14.688367 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:00:14.688342    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5bec735_88e4_4b14_899d_5e8158d2638c.slice/crio-5fc5aabc0787df693841cb81e1ac362ddc885755a9a28012f704ad0138a064b4 WatchSource:0}: Error finding container 5fc5aabc0787df693841cb81e1ac362ddc885755a9a28012f704ad0138a064b4: Status 404 returned error can't find the container with id 5fc5aabc0787df693841cb81e1ac362ddc885755a9a28012f704ad0138a064b4
Oct 30 01:00:14.690810 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.690780    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:00:14.690882 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.690846    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:00:14.690882 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:14.690875    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:00:15.029991 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:15.029957    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq" event={"ID":"b5bec735-88e4-4b14-899d-5e8158d2638c","Type":"ContainerStarted","Data":"659b6a3e82e056ddf66f729280fa5d8c18442a0b40fe385c211306e4ee04ac32"}
Oct 30 01:00:15.030174 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:15.030010    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq" event={"ID":"b5bec735-88e4-4b14-899d-5e8158d2638c","Type":"ContainerStarted","Data":"5fc5aabc0787df693841cb81e1ac362ddc885755a9a28012f704ad0138a064b4"}
Oct 30 01:00:15.074735 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:15.074680    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq" podStartSLOduration=1.074665689 podStartE2EDuration="1.074665689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:15.071636885 +0000 UTC m=+1127.409738795" watchObservedRunningTime="2026-10-30 01:00:15.074665689 +0000 UTC m=+1127.412767598"
Oct 30 01:00:15.366153 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:15.365699    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:15.382008 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:15.381965    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8s5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 01:00:15.382285 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:15.382031    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq" podUID="b5bec735-88e4-4b14-899d-5e8158d2638c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 01:00:16.381095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:16.380476    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8s5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 01:00:16.381095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:16.380553    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq" podUID="b5bec735-88e4-4b14-899d-5e8158d2638c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 01:00:17.381220 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:17.373480    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8s5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 01:00:17.381220 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:17.373543    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq" podUID="b5bec735-88e4-4b14-899d-5e8158d2638c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 01:00:18.367528 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:18.367487    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8s5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 01:00:18.367792 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:18.367764    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq" podUID="b5bec735-88e4-4b14-899d-5e8158d2638c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 01:00:19.040640 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:00:19.040605    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:00:19.381286 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:19.381204    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8s5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 01:00:19.381286 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:19.381261    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq" podUID="b5bec735-88e4-4b14-899d-5e8158d2638c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 01:00:20.381648 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:20.381606    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8s5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 01:00:20.382280 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:20.382243    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq" podUID="b5bec735-88e4-4b14-899d-5e8158d2638c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 01:00:21.366277 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:21.366183    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8s5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 01:00:21.366277 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:21.366256    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq" podUID="b5bec735-88e4-4b14-899d-5e8158d2638c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 01:00:22.366460 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:22.366423    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8s5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 01:00:22.366832 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:22.366490    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq" podUID="b5bec735-88e4-4b14-899d-5e8158d2638c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 01:00:23.370722 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:23.370691    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:23.371149 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:23.371099    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:23.371831 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:23.371807    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:00:26.228687 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.228649    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"]
Oct 30 01:00:26.243493 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.243460    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.274522 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.274489    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"]
Oct 30 01:00:26.345547 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.345516    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.345547 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.345557    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.345784 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.345579    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.345784 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.345687    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.345784 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.345786    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbzs2\" (UniqueName: \"kubernetes.io/projected/eab47e5b-e868-483e-92a2-e3414cc1dda2-kube-api-access-kbzs2\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.345920 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.345895    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab47e5b-e868-483e-92a2-e3414cc1dda2-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.345961 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.345943    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.346014 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.346002    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.346135 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.346064    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.447720 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.447685    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.447918 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.447734    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbzs2\" (UniqueName: \"kubernetes.io/projected/eab47e5b-e868-483e-92a2-e3414cc1dda2-kube-api-access-kbzs2\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.447918 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.447801    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab47e5b-e868-483e-92a2-e3414cc1dda2-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.447918 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.447865    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.447918 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.447919    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.448200 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.447960    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.448200 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.448104    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.448200 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.448128    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.448200 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.448158    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.448404 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.448322    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.448404 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.448372    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.448599 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.448575    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.448662 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.448611    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab47e5b-e868-483e-92a2-e3414cc1dda2-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.448662 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.448648    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.450201 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.450176    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.450660 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.450641    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.469452 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.469429    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.469561 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.469514    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbzs2\" (UniqueName: \"kubernetes.io/projected/eab47e5b-e868-483e-92a2-e3414cc1dda2-kube-api-access-kbzs2\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.574360 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.574271    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:26.923763 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.923737    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"]
Oct 30 01:00:26.925099 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:00:26.925046    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab47e5b_e868_483e_92a2_e3414cc1dda2.slice/crio-38f6a833e62861e0e98820ec2290767679313d4bc9f4693ab17ca37364753753 WatchSource:0}: Error finding container 38f6a833e62861e0e98820ec2290767679313d4bc9f4693ab17ca37364753753: Status 404 returned error can't find the container with id 38f6a833e62861e0e98820ec2290767679313d4bc9f4693ab17ca37364753753
Oct 30 01:00:26.927519 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.927485    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:00:26.927625 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.927553    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:00:26.927625 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:26.927582    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:00:27.142064 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:27.142027    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz" event={"ID":"eab47e5b-e868-483e-92a2-e3414cc1dda2","Type":"ContainerStarted","Data":"1e4c6877bd214bf6f466de8c25bde514f13f7614ad44dca3959e3b40fb12950a"}
Oct 30 01:00:27.142305 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:27.142107    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz" event={"ID":"eab47e5b-e868-483e-92a2-e3414cc1dda2","Type":"ContainerStarted","Data":"38f6a833e62861e0e98820ec2290767679313d4bc9f4693ab17ca37364753753"}
Oct 30 01:00:27.184176 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:27.183945    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz" podStartSLOduration=1.18391513 podStartE2EDuration="1.18391513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:27.18294578 +0000 UTC m=+1139.521047680" watchObservedRunningTime="2026-10-30 01:00:27.18391513 +0000 UTC m=+1139.522017063"
Oct 30 01:00:27.575017 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:27.574989    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:28.583189 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:28.583149    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 01:00:28.583591 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:28.583213    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz" podUID="eab47e5b-e868-483e-92a2-e3414cc1dda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 01:00:29.090633 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.090600    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"]
Oct 30 01:00:29.096432 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.096404    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.128100 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.126973    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"]
Oct 30 01:00:29.135523 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.134146    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"]
Oct 30 01:00:29.148122 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.146881    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.201757 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.201719    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"]
Oct 30 01:00:29.206378 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.206341    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.206551 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.206399    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.206551 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.206463    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.206551 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.206507    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.206725 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.206563    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.206725 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.206674    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.206841 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.206804    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpcst\" (UniqueName: \"kubernetes.io/projected/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-kube-api-access-lpcst\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.206896 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.206842    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.206896 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.206887    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.308194 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.308153    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92bdf87-b7d6-4060-9615-70ae77214bc0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.308389 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.308255    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.308389 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.308307    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.308498 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.308391    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.308498 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.308413    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpcst\" (UniqueName: \"kubernetes.io/projected/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-kube-api-access-lpcst\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.308498 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.308450    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.308627 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.308566    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.308627 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.308615    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.308729 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.308697    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.308781 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.308738    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.308837 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.308778    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.308837 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.308806    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxkl8\" (UniqueName: \"kubernetes.io/projected/e92bdf87-b7d6-4060-9615-70ae77214bc0-kube-api-access-lxkl8\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.308936 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.308853    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.308987 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.308964    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.309124 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.309044    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.309124 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.309115    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.309254 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.309135    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.309425 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.309404    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.309493 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.309461    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.309659 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.309637    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.309659 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.309639    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.309775 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.309702    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.309775 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.309706    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.311088 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.311047    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.311838 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.311816    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.330051 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.330013    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.330215 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.330158    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpcst\" (UniqueName: \"kubernetes.io/projected/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-kube-api-access-lpcst\") pod \"maas-default-gateway-openshift-default-fcb75c45f-grnvw\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.410642 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.410546    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92bdf87-b7d6-4060-9615-70ae77214bc0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.410642 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.410603    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.410846 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.410674    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.410846 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.410722    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.410846 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.410742    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxkl8\" (UniqueName: \"kubernetes.io/projected/e92bdf87-b7d6-4060-9615-70ae77214bc0-kube-api-access-lxkl8\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.410846 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.410771    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.410846 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.410820    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.410999 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.410846    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.410999 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.410888    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.411185 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.411163    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.411276 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.411221    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.411370 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.411351    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.411491 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.411461    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92bdf87-b7d6-4060-9615-70ae77214bc0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.411616 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.411545    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.411676 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.411647    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:29.413282 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.413260    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.413641 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.413624    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.433267 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.433237    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.433397 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.433279    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxkl8\" (UniqueName: \"kubernetes.io/projected/e92bdf87-b7d6-4060-9615-70ae77214bc0-kube-api-access-lxkl8\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zct5s\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.525335 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.525290    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:29.592011 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.591298    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 01:00:29.592011 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.591367    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz" podUID="eab47e5b-e868-483e-92a2-e3414cc1dda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 01:00:29.794981 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:00:29.794945    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370c4c48_476a_45b1_bb4b_09f1d92d0fa1.slice/crio-b8aea7345cf557a3e0b117fc8c65ab4e60a6b0a2d5e1ceb0f795175892b7c5ba WatchSource:0}: Error finding container b8aea7345cf557a3e0b117fc8c65ab4e60a6b0a2d5e1ceb0f795175892b7c5ba: Status 404 returned error can't find the container with id b8aea7345cf557a3e0b117fc8c65ab4e60a6b0a2d5e1ceb0f795175892b7c5ba
Oct 30 01:00:29.798441 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.798416    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"]
Oct 30 01:00:29.799368 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.799336    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:00:29.799451 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.799406    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:00:29.799451 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.799435    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:00:29.900251 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.900224    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"]
Oct 30 01:00:29.902271 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:00:29.902246    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92bdf87_b7d6_4060_9615_70ae77214bc0.slice/crio-d4b7fc3593115050a7701bf19ace7603d52b83e20c412c6b2127b3cc73a11591 WatchSource:0}: Error finding container d4b7fc3593115050a7701bf19ace7603d52b83e20c412c6b2127b3cc73a11591: Status 404 returned error can't find the container with id d4b7fc3593115050a7701bf19ace7603d52b83e20c412c6b2127b3cc73a11591
Oct 30 01:00:29.904888 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.904852    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:00:29.904966 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.904925    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:00:29.904966 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:29.904956    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:00:30.214294 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:30.214259    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw" event={"ID":"370c4c48-476a-45b1-bb4b-09f1d92d0fa1","Type":"ContainerStarted","Data":"fa4f560e8b55bcd008e187a3600d54b82f0e1cb7582a6f4381c0c1b96b0dd6e2"}
Oct 30 01:00:30.214488 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:30.214318    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw" event={"ID":"370c4c48-476a-45b1-bb4b-09f1d92d0fa1","Type":"ContainerStarted","Data":"b8aea7345cf557a3e0b117fc8c65ab4e60a6b0a2d5e1ceb0f795175892b7c5ba"}
Oct 30 01:00:30.216724 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:30.216697    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s" event={"ID":"e92bdf87-b7d6-4060-9615-70ae77214bc0","Type":"ContainerStarted","Data":"f87e5624c1595c3e1095260cb5fa030036ab26d533f8eaddf333d1a776c477a6"}
Oct 30 01:00:30.216863 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:30.216744    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s" event={"ID":"e92bdf87-b7d6-4060-9615-70ae77214bc0","Type":"ContainerStarted","Data":"d4b7fc3593115050a7701bf19ace7603d52b83e20c412c6b2127b3cc73a11591"}
Oct 30 01:00:30.263359 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:30.263248    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw" podStartSLOduration=1.263230351 podStartE2EDuration="1.263230351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:30.26156046 +0000 UTC m=+1142.599662382" watchObservedRunningTime="2026-10-30 01:00:30.263230351 +0000 UTC m=+1142.601332261"
Oct 30 01:00:30.321023 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:30.320956    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s" podStartSLOduration=1.320939968 podStartE2EDuration="1.320939968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:30.318452329 +0000 UTC m=+1142.656554253" watchObservedRunningTime="2026-10-30 01:00:30.320939968 +0000 UTC m=+1142.659041877"
Oct 30 01:00:30.412723 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:30.412692    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:30.414427 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:30.414405    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-grnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 01:00:30.414504 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:30.414457    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw" podUID="370c4c48-476a-45b1-bb4b-09f1d92d0fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 01:00:30.526386 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:30.526299    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:30.579100 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:30.575403    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 01:00:30.579100 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:30.575481    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz" podUID="eab47e5b-e868-483e-92a2-e3414cc1dda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 01:00:31.415911 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:31.415445    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-grnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 01:00:31.415911 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:31.415523    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw" podUID="370c4c48-476a-45b1-bb4b-09f1d92d0fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 01:00:31.544092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:31.543233    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zct5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 01:00:31.544092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:31.543297    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s" podUID="e92bdf87-b7d6-4060-9615-70ae77214bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 01:00:31.584096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:31.583410    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 01:00:31.584096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:31.583468    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz" podUID="eab47e5b-e868-483e-92a2-e3414cc1dda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 01:00:32.440096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:32.434317    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-grnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 01:00:32.440096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:32.434471    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw" podUID="370c4c48-476a-45b1-bb4b-09f1d92d0fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 01:00:32.533107 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:32.530258    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zct5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 01:00:32.533107 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:32.530321    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s" podUID="e92bdf87-b7d6-4060-9615-70ae77214bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 01:00:32.576577 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:32.576539    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 01:00:32.576948 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:32.576831    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz" podUID="eab47e5b-e868-483e-92a2-e3414cc1dda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 01:00:33.415576 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:33.415535    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-grnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 01:00:33.415887 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:33.415849    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw" podUID="370c4c48-476a-45b1-bb4b-09f1d92d0fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 01:00:33.528661 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:33.528625    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zct5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 01:00:33.529282 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:33.529236    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s" podUID="e92bdf87-b7d6-4060-9615-70ae77214bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 01:00:33.576553 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:33.576504    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 01:00:33.576830 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:33.576803    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz" podUID="eab47e5b-e868-483e-92a2-e3414cc1dda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 01:00:34.413511 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:34.413463    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-grnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 01:00:34.413800 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:34.413768    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw" podUID="370c4c48-476a-45b1-bb4b-09f1d92d0fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 01:00:34.529500 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:34.529461    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zct5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 01:00:34.530131 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:34.530098    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s" podUID="e92bdf87-b7d6-4060-9615-70ae77214bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 01:00:34.576960 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:34.576904    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 01:00:34.577299 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:34.577253    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz" podUID="eab47e5b-e868-483e-92a2-e3414cc1dda2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 01:00:35.420096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:35.412512    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-grnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 01:00:35.420096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:35.412582    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw" podUID="370c4c48-476a-45b1-bb4b-09f1d92d0fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 01:00:35.528041 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:35.528007    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zct5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 01:00:35.528354 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:35.528321    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s" podUID="e92bdf87-b7d6-4060-9615-70ae77214bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 01:00:35.659103 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:35.655314    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29" podUID="e0ab9bcb-f48a-404b-b787-9f8c43b58722" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 01:00:35.665113 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:35.663171    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj" podUID="63a8525d-9951-4f19-b124-62e4fa91c6b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:36.175413 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:36.174160    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj" podUID="8b267884-baae-4ad3-a414-82467f7188b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:36.221090 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:36.205665    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll" podUID="2388a847-21ff-46dd-8b45-4183a2448cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:36.226184 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:36.226143    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz" podUID="86e98e6a-b1cf-4d08-b807-26397cd1b4a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 30 01:00:36.425605 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:36.425514    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-grnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 01:00:36.425873 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:36.425836    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw" podUID="370c4c48-476a-45b1-bb4b-09f1d92d0fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 01:00:36.478240 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:36.478196    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:36.480854 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:36.480832    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:36.491477 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:36.491445    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:00:36.550265 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:36.550193    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh" podUID="6ba3a068-e4af-4bd1-b007-2f73f455f65c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:36.636400 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:36.636365    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zct5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 01:00:36.636672 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:36.636639    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s" podUID="e92bdf87-b7d6-4060-9615-70ae77214bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 01:00:36.657870 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:36.657828    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29" podUID="e0ab9bcb-f48a-404b-b787-9f8c43b58722" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 01:00:36.662905 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:36.662876    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 01:00:36.697382 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:36.697265    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv" podUID="3c545315-888c-4418-8348-b06a0080a599" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:37.182097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:37.181200    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj" podUID="8b267884-baae-4ad3-a414-82467f7188b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:37.182097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:37.181992    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 01:00:37.244095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:37.242306    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll" podUID="2388a847-21ff-46dd-8b45-4183a2448cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:37.244095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:37.242953    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 01:00:37.461475 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:37.457473    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-grnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 01:00:37.461475 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:37.457538    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw" podUID="370c4c48-476a-45b1-bb4b-09f1d92d0fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 01:00:37.568095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:37.560416    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh" podUID="6ba3a068-e4af-4bd1-b007-2f73f455f65c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Oct 30 01:00:37.588238 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:37.569262    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 01:00:37.588238 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:37.570802    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zct5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 01:00:37.588238 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:37.570853    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s" podUID="e92bdf87-b7d6-4060-9615-70ae77214bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 01:00:37.699432 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:37.699377    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29" podUID="e0ab9bcb-f48a-404b-b787-9f8c43b58722" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:37.740151 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:37.708355    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv" podUID="3c545315-888c-4418-8348-b06a0080a599" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 01:00:37.778097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:37.744368    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 01:00:38.259095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:38.256203    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj" podUID="8b267884-baae-4ad3-a414-82467f7188b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:38.317103 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:38.315684    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll" podUID="2388a847-21ff-46dd-8b45-4183a2448cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:38.346589 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:38.346541    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz" podUID="86e98e6a-b1cf-4d08-b807-26397cd1b4a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:38.352669 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:38.352633    4864 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj" podUID="63a8525d-9951-4f19-b124-62e4fa91c6b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:38.432448 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:38.422703    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-grnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 01:00:38.432714 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:38.432680    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw" podUID="370c4c48-476a-45b1-bb4b-09f1d92d0fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 01:00:38.492293 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:38.492260    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 01:00:38.530153 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:38.529356    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zct5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 01:00:38.530153 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:38.529439    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s" podUID="e92bdf87-b7d6-4060-9615-70ae77214bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 01:00:38.563311 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:38.563268    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh" podUID="6ba3a068-e4af-4bd1-b007-2f73f455f65c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:38.711107 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:38.710477    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29" podUID="e0ab9bcb-f48a-404b-b787-9f8c43b58722" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:38.737715 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:38.736602    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj" podUID="63a8525d-9951-4f19-b124-62e4fa91c6b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:38.745513 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:38.745466    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv" podUID="3c545315-888c-4418-8348-b06a0080a599" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:38.829798 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:38.829715    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 01:00:38.838172 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:38.831577    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 01:00:38.888695 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:38.888665    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 01:00:38.920631 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:38.920552    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 01:00:39.412564 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:39.412532    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-grnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 01:00:39.412739 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:39.412582    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw" podUID="370c4c48-476a-45b1-bb4b-09f1d92d0fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 01:00:39.526797 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:39.526758    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zct5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 01:00:39.526993 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:39.526827    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s" podUID="e92bdf87-b7d6-4060-9615-70ae77214bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 01:00:41.355348 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:41.355315    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:41.356152 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:41.356131    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:41.356801 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:41.356780    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:00:41.553279 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:41.553232    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zct5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:00:41.553461 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:41.553318    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s" podUID="e92bdf87-b7d6-4060-9615-70ae77214bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:41.645858 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:41.645474    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:41.647054 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:41.647004    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:41.647816 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:41.647798    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:00:46.154058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:46.154021    4864 generic.go:356] "Generic (PLEG): container finished" podID="8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab" containerID="70f8fbcb166f228c99e51626618b0fead6c49eb15873f416e2daca3206da0721" exitCode=0
Oct 30 01:00:46.154509 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:46.154098    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp" event={"ID":"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab","Type":"ContainerDied","Data":"70f8fbcb166f228c99e51626618b0fead6c49eb15873f416e2daca3206da0721"}
Oct 30 01:00:46.154555 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:46.154541    4864 scope.go:122] "RemoveContainer" containerID="70f8fbcb166f228c99e51626618b0fead6c49eb15873f416e2daca3206da0721"
Oct 30 01:00:46.156013 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:46.155989    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:00:46.156092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:46.156055    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:00:46.156134 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:46.156107    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:00:47.200533 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:47.199273    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp" event={"ID":"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab","Type":"ContainerStarted","Data":"66c37ca618ebe48ff2e71236435fb4211792c9605a8208fb94a17155247d01c7"}
Oct 30 01:00:47.385423 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:47.385389    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 01:00:47.385764 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:47.385741    4864 patch_prober.go:29] interesting pod/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 01:00:47.385810 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:47.385793    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp" podUID="8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 01:00:48.385629 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:48.385593    4864 patch_prober.go:29] interesting pod/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 01:00:48.386033 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:48.385663    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp" podUID="8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 01:00:49.041733 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:00:49.041694    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:00:49.385656 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:49.385566    4864 patch_prober.go:29] interesting pod/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 01:00:49.385656 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:49.385621    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp" podUID="8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 01:00:50.385795 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:50.385757    4864 patch_prober.go:29] interesting pod/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 01:00:50.386281 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:50.385821    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp" podUID="8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 01:00:51.385708 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:51.385666    4864 patch_prober.go:29] interesting pod/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 01:00:51.385923 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:51.385742    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp" podUID="8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 01:00:52.390088 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:52.390040    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 01:00:52.390565 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:52.390367    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 01:00:52.391124 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:00:52.391107    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 01:01:19.046060 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:01:19.046019    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:01:23.630223 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.630179    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"]
Oct 30 01:01:23.633873 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.633849    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.650352 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.650323    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8363a40b-openshift-default-dockercfg-s5kzj\""
Oct 30 01:01:23.666939 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.666909    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"]
Oct 30 01:01:23.740378 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.740342    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-credential-socket\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.740378 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.740377    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-data\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.740586 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.740503    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-workload-socket\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.740586 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.740575    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-envoy\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.740678 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.740666    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-podinfo\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.740713 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.740688    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04553b1e-6cf7-498d-8fab-b282992c70f3-istiod-ca-cert\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.740799 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.740783    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r256q\" (UniqueName: \"kubernetes.io/projected/04553b1e-6cf7-498d-8fab-b282992c70f3-kube-api-access-r256q\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.740892 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.740880    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-token\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.740926 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.740916    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-workload-certs\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.842771 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.842737    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r256q\" (UniqueName: \"kubernetes.io/projected/04553b1e-6cf7-498d-8fab-b282992c70f3-kube-api-access-r256q\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.842944 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.842879    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-token\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.842944 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.842932    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-workload-certs\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.843029 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.842978    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-credential-socket\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.843029 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.843001    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-data\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.843170 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.843063    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-workload-socket\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.843170 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.843163    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-envoy\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.843304 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.843287    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-podinfo\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.843363 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.843314    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04553b1e-6cf7-498d-8fab-b282992c70f3-istiod-ca-cert\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.843470 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.843450    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-workload-certs\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.843538 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.843493    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-credential-socket\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.843600 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.843565    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-data\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.843600 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.843576    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-workload-socket\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.844025 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.844004    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04553b1e-6cf7-498d-8fab-b282992c70f3-istiod-ca-cert\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.845613 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.845587    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-envoy\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.845749 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.845729    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-podinfo\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.866907 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.866880    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-token\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.867117 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.867096    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r256q\" (UniqueName: \"kubernetes.io/projected/04553b1e-6cf7-498d-8fab-b282992c70f3-kube-api-access-r256q\") pod \"e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") " pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:23.947737 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:23.947658    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:24.260606 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:24.260577    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"]
Oct 30 01:01:24.262200 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:01:24.262175    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04553b1e_6cf7_498d_8fab_b282992c70f3.slice/crio-7d19a720a9d6cb39f6d1b943ffeea8bfa941d14a3ea8da733ea0eee1801b7f4d WatchSource:0}: Error finding container 7d19a720a9d6cb39f6d1b943ffeea8bfa941d14a3ea8da733ea0eee1801b7f4d: Status 404 returned error can't find the container with id 7d19a720a9d6cb39f6d1b943ffeea8bfa941d14a3ea8da733ea0eee1801b7f4d
Oct 30 01:01:24.264728 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:24.264697    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:01:24.264821 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:24.264762    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:01:24.264821 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:24.264808    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:01:24.602607 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:24.602495    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss" event={"ID":"04553b1e-6cf7-498d-8fab-b282992c70f3","Type":"ContainerStarted","Data":"a15c95a0cb4a1cf71345b62150c4268c1b70aaf61c7f6796d721f8b9e4474061"}
Oct 30 01:01:24.602915 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:24.602607    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss" event={"ID":"04553b1e-6cf7-498d-8fab-b282992c70f3","Type":"ContainerStarted","Data":"7d19a720a9d6cb39f6d1b943ffeea8bfa941d14a3ea8da733ea0eee1801b7f4d"}
Oct 30 01:01:24.666915 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:24.666863    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss" podStartSLOduration=1.666848423 podStartE2EDuration="1.666848423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:24.648708616 +0000 UTC m=+1196.986810531" watchObservedRunningTime="2026-10-30 01:01:24.666848423 +0000 UTC m=+1197.004950332"
Oct 30 01:01:24.954162 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:24.952188    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:25.949452 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:25.949413    4864 patch_prober.go:29] interesting pod/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 01:01:25.949865 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:25.949479    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss" podUID="04553b1e-6cf7-498d-8fab-b282992c70f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 01:01:26.948924 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:26.948884    4864 patch_prober.go:29] interesting pod/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 01:01:26.949153 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:26.948958    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss" podUID="04553b1e-6cf7-498d-8fab-b282992c70f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 01:01:27.939665 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:27.939631    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:01:27.940067 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:27.939907    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:01:27.948067 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:27.948030    4864 patch_prober.go:29] interesting pod/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 01:01:27.948234 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:27.948101    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss" podUID="04553b1e-6cf7-498d-8fab-b282992c70f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 01:01:28.948238 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:28.948206    4864 patch_prober.go:29] interesting pod/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 01:01:28.948645 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:28.948265    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss" podUID="04553b1e-6cf7-498d-8fab-b282992c70f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 01:01:29.948763 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:29.948728    4864 patch_prober.go:29] interesting pod/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 01:01:29.949212 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:29.948780    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss" podUID="04553b1e-6cf7-498d-8fab-b282992c70f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 01:01:31.038613 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:31.038573    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:31.039148 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:31.039042    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:31.040127 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:01:31.040104    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:01:49.042047 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:01:49.041958    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:02:19.040563 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:02:19.040529    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:02:24.652193 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.652157    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"]
Oct 30 01:02:24.657270 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.657243    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.671173 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.671127    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9b2c3f-openshift-default-dockercfg-4fnr9\""
Oct 30 01:02:24.688471 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.688441    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"]
Oct 30 01:02:24.815096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.814381    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istiod-ca-cert\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.815096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.814432    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-envoy\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.836900 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.836638    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm2bq\" (UniqueName: \"kubernetes.io/projected/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-kube-api-access-jm2bq\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.836900 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.836724    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-credential-socket\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.836900 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.836760    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-podinfo\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.836900 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.836801    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-data\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.836900 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.836858    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-workload-certs\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.837287 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.836941    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-workload-socket\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.837287 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.837053    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-token\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.938979 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.938903    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istiod-ca-cert\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.938979 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.938937    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-envoy\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.939260 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.939039    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm2bq\" (UniqueName: \"kubernetes.io/projected/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-kube-api-access-jm2bq\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.939260 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.939106    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-credential-socket\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.939260 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.939130    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-podinfo\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.939413 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.939253    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-data\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.939413 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.939326    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-workload-certs\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.939413 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.939408    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-workload-socket\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.939577 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.939496    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-credential-socket\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.939577 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.939532    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-token\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.939688 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.939654    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-data\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.939773 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.939746    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-workload-certs\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.939839 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.939791    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istiod-ca-cert\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.940188 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.940165    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-workload-socket\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.941346 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.941323    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-envoy\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.941711 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.941691    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-podinfo\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.960884 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.960855    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-token\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.961126 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.961095    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm2bq\" (UniqueName: \"kubernetes.io/projected/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-kube-api-access-jm2bq\") pod \"e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") " pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:24.974139 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:24.973887    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:25.309945 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:25.309807    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"]
Oct 30 01:02:25.313292 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:02:25.313257    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27f9baa_b6fc_4d5b_9c49_56e93c2a703b.slice/crio-9f5be449f4fa0cfe9d96d7878e5a572a874d40765679877b5b32e1d64f8eb46e WatchSource:0}: Error finding container 9f5be449f4fa0cfe9d96d7878e5a572a874d40765679877b5b32e1d64f8eb46e: Status 404 returned error can't find the container with id 9f5be449f4fa0cfe9d96d7878e5a572a874d40765679877b5b32e1d64f8eb46e
Oct 30 01:02:25.315656 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:25.315623    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:02:25.315738 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:25.315694    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:02:25.315738 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:25.315726    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:02:26.138593 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:26.138545    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt" event={"ID":"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b","Type":"ContainerStarted","Data":"c6d4f8300c978be5d76ad8699a85f890b7f3f23b49a3f3cbad9965ba4a19d487"}
Oct 30 01:02:26.139056 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:26.138602    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt" event={"ID":"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b","Type":"ContainerStarted","Data":"9f5be449f4fa0cfe9d96d7878e5a572a874d40765679877b5b32e1d64f8eb46e"}
Oct 30 01:02:26.193065 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:26.193015    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt" podStartSLOduration=2.193000521 podStartE2EDuration="2.193000521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:26.189185599 +0000 UTC m=+1258.527287508" watchObservedRunningTime="2026-10-30 01:02:26.193000521 +0000 UTC m=+1258.531102430"
Oct 30 01:02:26.367217 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:26.367188    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9"]
Oct 30 01:02:26.367421 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:26.367397    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9" podUID="8b21e6da-169c-494d-8afa-cb8288ebdeb0" containerName="maas-api" containerID="cri-o://f0b8cb847d076248d7fed0621a90105531c1812d63b4f1eac734b359cbde97d8" gracePeriod=30
Oct 30 01:02:26.671994 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:26.671926    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9"
Oct 30 01:02:26.773566 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:26.773537    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b21e6da-169c-494d-8afa-cb8288ebdeb0-kube-api-access-x5gc7\" (UniqueName: \"kubernetes.io/projected/8b21e6da-169c-494d-8afa-cb8288ebdeb0-kube-api-access-x5gc7\") pod \"8b21e6da-169c-494d-8afa-cb8288ebdeb0\" (UID: \"8b21e6da-169c-494d-8afa-cb8288ebdeb0\") "
Oct 30 01:02:26.773731 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:26.773667    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b21e6da-169c-494d-8afa-cb8288ebdeb0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b21e6da-169c-494d-8afa-cb8288ebdeb0-maas-api-tls\") pod \"8b21e6da-169c-494d-8afa-cb8288ebdeb0\" (UID: \"8b21e6da-169c-494d-8afa-cb8288ebdeb0\") "
Oct 30 01:02:26.773731 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:26.773703    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b21e6da-169c-494d-8afa-cb8288ebdeb0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b21e6da-169c-494d-8afa-cb8288ebdeb0-metrics-certs\") pod \"8b21e6da-169c-494d-8afa-cb8288ebdeb0\" (UID: \"8b21e6da-169c-494d-8afa-cb8288ebdeb0\") "
Oct 30 01:02:26.773863 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:26.773729    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b21e6da-169c-494d-8afa-cb8288ebdeb0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b21e6da-169c-494d-8afa-cb8288ebdeb0-openshift-service-ca\") pod \"8b21e6da-169c-494d-8afa-cb8288ebdeb0\" (UID: \"8b21e6da-169c-494d-8afa-cb8288ebdeb0\") "
Oct 30 01:02:26.774198 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:26.774162    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b21e6da-169c-494d-8afa-cb8288ebdeb0-openshift-service-ca" pod "8b21e6da-169c-494d-8afa-cb8288ebdeb0" (UID: "8b21e6da-169c-494d-8afa-cb8288ebdeb0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:02:26.776224 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:26.774808    4864 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b21e6da-169c-494d-8afa-cb8288ebdeb0-openshift-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:02:26.776454 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:26.776423    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b21e6da-169c-494d-8afa-cb8288ebdeb0-metrics-certs" pod "8b21e6da-169c-494d-8afa-cb8288ebdeb0" (UID: "8b21e6da-169c-494d-8afa-cb8288ebdeb0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:02:26.776531 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:26.776506    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b21e6da-169c-494d-8afa-cb8288ebdeb0-kube-api-access-x5gc7" pod "8b21e6da-169c-494d-8afa-cb8288ebdeb0" (UID: "8b21e6da-169c-494d-8afa-cb8288ebdeb0"). InnerVolumeSpecName "kube-api-access-x5gc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:26.777494 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:26.777467    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b21e6da-169c-494d-8afa-cb8288ebdeb0-maas-api-tls" pod "8b21e6da-169c-494d-8afa-cb8288ebdeb0" (UID: "8b21e6da-169c-494d-8afa-cb8288ebdeb0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:02:26.887123 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:26.884611    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b21e6da-169c-494d-8afa-cb8288ebdeb0-maas-api-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:02:26.887123 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:26.884647    4864 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b21e6da-169c-494d-8afa-cb8288ebdeb0-metrics-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:02:26.887123 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:26.884665    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5gc7\" (UniqueName: \"kubernetes.io/projected/8b21e6da-169c-494d-8afa-cb8288ebdeb0-kube-api-access-x5gc7\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:02:26.974602 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:26.974569    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:26.983133 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:26.983064    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:27.148900 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:27.148858    4864 generic.go:356] "Generic (PLEG): container finished" podID="8b21e6da-169c-494d-8afa-cb8288ebdeb0" containerID="f0b8cb847d076248d7fed0621a90105531c1812d63b4f1eac734b359cbde97d8" exitCode=0
Oct 30 01:02:27.149425 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:27.148950    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9" event={"ID":"8b21e6da-169c-494d-8afa-cb8288ebdeb0","Type":"ContainerDied","Data":"f0b8cb847d076248d7fed0621a90105531c1812d63b4f1eac734b359cbde97d8"}
Oct 30 01:02:27.149425 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:27.149052    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9" event={"ID":"8b21e6da-169c-494d-8afa-cb8288ebdeb0","Type":"ContainerDied","Data":"ceb9326bbfe487c4b2b57aed85ad7ebd10944245b25f5ed69a5a534feba8a3d4"}
Oct 30 01:02:27.149425 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:27.149123    4864 scope.go:122] "RemoveContainer" containerID="f0b8cb847d076248d7fed0621a90105531c1812d63b4f1eac734b359cbde97d8"
Oct 30 01:02:27.149425 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:27.148964    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9"
Oct 30 01:02:27.150049 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:27.150031    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:27.151211 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:27.151185    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:02:27.167241 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:27.167215    4864 scope.go:122] "RemoveContainer" containerID="f0b8cb847d076248d7fed0621a90105531c1812d63b4f1eac734b359cbde97d8"
Oct 30 01:02:27.167717 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:02:27.167682    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0b8cb847d076248d7fed0621a90105531c1812d63b4f1eac734b359cbde97d8\": container with ID starting with f0b8cb847d076248d7fed0621a90105531c1812d63b4f1eac734b359cbde97d8 not found: ID does not exist" containerID="f0b8cb847d076248d7fed0621a90105531c1812d63b4f1eac734b359cbde97d8"
Oct 30 01:02:27.167962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:27.167919    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0b8cb847d076248d7fed0621a90105531c1812d63b4f1eac734b359cbde97d8"} err="failed to get container status \"f0b8cb847d076248d7fed0621a90105531c1812d63b4f1eac734b359cbde97d8\": rpc error: code = NotFound desc = could not find container \"f0b8cb847d076248d7fed0621a90105531c1812d63b4f1eac734b359cbde97d8\": container with ID starting with f0b8cb847d076248d7fed0621a90105531c1812d63b4f1eac734b359cbde97d8 not found: ID does not exist"
Oct 30 01:02:27.213376 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:27.213341    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9"]
Oct 30 01:02:27.230183 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:27.219637    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8363a40b-5dbd55b679-tjwp9"]
Oct 30 01:02:27.794427 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:27.794393    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b21e6da-169c-494d-8afa-cb8288ebdeb0" path="/var/lib/kubelet/pods/8b21e6da-169c-494d-8afa-cb8288ebdeb0/volumes"
Oct 30 01:02:38.193996 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:38.193957    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"]
Oct 30 01:02:38.194675 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:38.194657    4864 cpu_manager.go:414] "containerMap: removing container" podUID="8b21e6da-169c-494d-8afa-cb8288ebdeb0" containerName="maas-api"
Oct 30 01:02:38.194675 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:38.194675    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b21e6da-169c-494d-8afa-cb8288ebdeb0" containerName="maas-api"
Oct 30 01:02:38.194881 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:38.194867    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b21e6da-169c-494d-8afa-cb8288ebdeb0" containerName="maas-api"
Oct 30 01:02:38.199858 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:38.199836    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"
Oct 30 01:02:38.216949 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:38.216921    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-9a8d85-serving-cert\""
Oct 30 01:02:38.235128 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:38.230273    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"]
Oct 30 01:02:38.281390 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:38.281352    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254d9154-d8b0-4058-9ebf-8963632489c7-maas-api-tls\") pod \"maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj\" (UID: \"254d9154-d8b0-4058-9ebf-8963632489c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"
Oct 30 01:02:38.281877 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:38.281846    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254d9154-d8b0-4058-9ebf-8963632489c7-openshift-service-ca\") pod \"maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj\" (UID: \"254d9154-d8b0-4058-9ebf-8963632489c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"
Oct 30 01:02:38.281991 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:38.281907    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5rbk\" (UniqueName: \"kubernetes.io/projected/254d9154-d8b0-4058-9ebf-8963632489c7-kube-api-access-k5rbk\") pod \"maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj\" (UID: \"254d9154-d8b0-4058-9ebf-8963632489c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"
Oct 30 01:02:38.281991 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:38.281938    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254d9154-d8b0-4058-9ebf-8963632489c7-metrics-certs\") pod \"maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj\" (UID: \"254d9154-d8b0-4058-9ebf-8963632489c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"
Oct 30 01:02:38.383842 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:38.383792    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254d9154-d8b0-4058-9ebf-8963632489c7-openshift-service-ca\") pod \"maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj\" (UID: \"254d9154-d8b0-4058-9ebf-8963632489c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"
Oct 30 01:02:38.384120 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:38.383862    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5rbk\" (UniqueName: \"kubernetes.io/projected/254d9154-d8b0-4058-9ebf-8963632489c7-kube-api-access-k5rbk\") pod \"maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj\" (UID: \"254d9154-d8b0-4058-9ebf-8963632489c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"
Oct 30 01:02:38.384120 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:38.383901    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254d9154-d8b0-4058-9ebf-8963632489c7-metrics-certs\") pod \"maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj\" (UID: \"254d9154-d8b0-4058-9ebf-8963632489c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"
Oct 30 01:02:38.384120 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:38.384038    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254d9154-d8b0-4058-9ebf-8963632489c7-maas-api-tls\") pod \"maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj\" (UID: \"254d9154-d8b0-4058-9ebf-8963632489c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"
Oct 30 01:02:38.384357 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:02:38.384337    4864 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-serving-cert: secret "maas-api-e2e-api-b-9a8d85-serving-cert" not found
Oct 30 01:02:38.384436 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:02:38.384423    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254d9154-d8b0-4058-9ebf-8963632489c7-maas-api-tls podName:254d9154-d8b0-4058-9ebf-8963632489c7 nodeName:}" failed. No retries permitted until 2026-10-30 01:02:38.884400842 +0000 UTC m=+1271.222502750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/254d9154-d8b0-4058-9ebf-8963632489c7-maas-api-tls") pod "maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj" (UID: "254d9154-d8b0-4058-9ebf-8963632489c7") : secret "maas-api-e2e-api-b-9a8d85-serving-cert" not found
Oct 30 01:02:38.385031 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:38.385003    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254d9154-d8b0-4058-9ebf-8963632489c7-openshift-service-ca\") pod \"maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj\" (UID: \"254d9154-d8b0-4058-9ebf-8963632489c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"
Oct 30 01:02:38.387598 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:38.387574    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254d9154-d8b0-4058-9ebf-8963632489c7-metrics-certs\") pod \"maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj\" (UID: \"254d9154-d8b0-4058-9ebf-8963632489c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"
Oct 30 01:02:38.409890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:38.409854    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5rbk\" (UniqueName: \"kubernetes.io/projected/254d9154-d8b0-4058-9ebf-8963632489c7-kube-api-access-k5rbk\") pod \"maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj\" (UID: \"254d9154-d8b0-4058-9ebf-8963632489c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"
Oct 30 01:02:38.895379 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:38.895343    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254d9154-d8b0-4058-9ebf-8963632489c7-maas-api-tls\") pod \"maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj\" (UID: \"254d9154-d8b0-4058-9ebf-8963632489c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"
Oct 30 01:02:38.903258 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:38.901485    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254d9154-d8b0-4058-9ebf-8963632489c7-maas-api-tls\") pod \"maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj\" (UID: \"254d9154-d8b0-4058-9ebf-8963632489c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"
Oct 30 01:02:39.114374 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:39.114322    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"
Oct 30 01:02:39.443875 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:39.443850    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"]
Oct 30 01:02:39.445588 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:02:39.445561    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254d9154_d8b0_4058_9ebf_8963632489c7.slice/crio-3d9d02216997a5cd4fa9d5df5b80e6b43ecfbbdf0ac52b1e4be739c43d78b5b3 WatchSource:0}: Error finding container 3d9d02216997a5cd4fa9d5df5b80e6b43ecfbbdf0ac52b1e4be739c43d78b5b3: Status 404 returned error can't find the container with id 3d9d02216997a5cd4fa9d5df5b80e6b43ecfbbdf0ac52b1e4be739c43d78b5b3
Oct 30 01:02:40.251928 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:40.251890    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj" event={"ID":"254d9154-d8b0-4058-9ebf-8963632489c7","Type":"ContainerStarted","Data":"378582ded3694f4fc3eb8aab61f28401e3e92151da3036308cb132810c7f52a9"}
Oct 30 01:02:40.252134 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:40.251952    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj" event={"ID":"254d9154-d8b0-4058-9ebf-8963632489c7","Type":"ContainerStarted","Data":"3d9d02216997a5cd4fa9d5df5b80e6b43ecfbbdf0ac52b1e4be739c43d78b5b3"}
Oct 30 01:02:40.252134 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:40.251988    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"
Oct 30 01:02:40.300958 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:40.300889    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj" podStartSLOduration=1.688885296 podStartE2EDuration="2.30086453s" totalImagesPullingTime="611.979234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:40.294420005 +0000 UTC m=+1272.632521912" watchObservedRunningTime="2026-10-30 01:02:40.30086453 +0000 UTC m=+1272.638966442"
Oct 30 01:02:46.263602 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:46.263571    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"
Oct 30 01:02:49.051803 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:02:49.051624    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:02:54.430844 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:54.428209    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8s5bq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:02:54.430844 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:54.428283    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq" podUID="b5bec735-88e4-4b14-899d-5e8158d2638c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:56.076097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:56.074212    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"]
Oct 30 01:02:56.076097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:02:56.074502    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp" podUID="8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab" containerName="istio-proxy" containerID="cri-o://66c37ca618ebe48ff2e71236435fb4211792c9605a8208fb94a17155247d01c7" gracePeriod=30
Oct 30 01:03:01.370449 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.370417    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 01:03:01.472950 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.472914    4864 generic.go:356] "Generic (PLEG): container finished" podID="8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab" containerID="66c37ca618ebe48ff2e71236435fb4211792c9605a8208fb94a17155247d01c7" exitCode=0
Oct 30 01:03:01.473196 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.473168    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp" event={"ID":"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab","Type":"ContainerDied","Data":"66c37ca618ebe48ff2e71236435fb4211792c9605a8208fb94a17155247d01c7"}
Oct 30 01:03:01.473283 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.473218    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp" event={"ID":"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab","Type":"ContainerDied","Data":"7dca8274f4c8c1ec7a966ff02e2cd894b48e004ffc1895da729cb0b4f4565eed"}
Oct 30 01:03:01.473283 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.473241    4864 scope.go:122] "RemoveContainer" containerID="66c37ca618ebe48ff2e71236435fb4211792c9605a8208fb94a17155247d01c7"
Oct 30 01:03:01.473411 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.473398    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"
Oct 30 01:03:01.487875 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.487827    4864 scope.go:122] "RemoveContainer" containerID="70f8fbcb166f228c99e51626618b0fead6c49eb15873f416e2daca3206da0721"
Oct 30 01:03:01.499461 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.499433    4864 scope.go:122] "RemoveContainer" containerID="66c37ca618ebe48ff2e71236435fb4211792c9605a8208fb94a17155247d01c7"
Oct 30 01:03:01.499812 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:03:01.499789    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66c37ca618ebe48ff2e71236435fb4211792c9605a8208fb94a17155247d01c7\": container with ID starting with 66c37ca618ebe48ff2e71236435fb4211792c9605a8208fb94a17155247d01c7 not found: ID does not exist" containerID="66c37ca618ebe48ff2e71236435fb4211792c9605a8208fb94a17155247d01c7"
Oct 30 01:03:01.499900 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.499820    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c37ca618ebe48ff2e71236435fb4211792c9605a8208fb94a17155247d01c7"} err="failed to get container status \"66c37ca618ebe48ff2e71236435fb4211792c9605a8208fb94a17155247d01c7\": rpc error: code = NotFound desc = could not find container \"66c37ca618ebe48ff2e71236435fb4211792c9605a8208fb94a17155247d01c7\": container with ID starting with 66c37ca618ebe48ff2e71236435fb4211792c9605a8208fb94a17155247d01c7 not found: ID does not exist"
Oct 30 01:03:01.499900 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.499838    4864 scope.go:122] "RemoveContainer" containerID="70f8fbcb166f228c99e51626618b0fead6c49eb15873f416e2daca3206da0721"
Oct 30 01:03:01.500182 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:03:01.500162    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f8fbcb166f228c99e51626618b0fead6c49eb15873f416e2daca3206da0721\": container with ID starting with 70f8fbcb166f228c99e51626618b0fead6c49eb15873f416e2daca3206da0721 not found: ID does not exist" containerID="70f8fbcb166f228c99e51626618b0fead6c49eb15873f416e2daca3206da0721"
Oct 30 01:03:01.500253 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.500188    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f8fbcb166f228c99e51626618b0fead6c49eb15873f416e2daca3206da0721"} err="failed to get container status \"70f8fbcb166f228c99e51626618b0fead6c49eb15873f416e2daca3206da0721\": rpc error: code = NotFound desc = could not find container \"70f8fbcb166f228c99e51626618b0fead6c49eb15873f416e2daca3206da0721\": container with ID starting with 70f8fbcb166f228c99e51626618b0fead6c49eb15873f416e2daca3206da0721 not found: ID does not exist"
Oct 30 01:03:01.512103 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.512051    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istiod-ca-cert\") pod \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") "
Oct 30 01:03:01.512231 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.512168    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-data\") pod \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") "
Oct 30 01:03:01.512273 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.512228    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-workload-certs\") pod \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") "
Oct 30 01:03:01.512321 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.512285    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-workload-socket\") pod \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") "
Oct 30 01:03:01.512428 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.512411    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-credential-socket\") pod \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") "
Oct 30 01:03:01.512490 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.512444    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-envoy\") pod \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") "
Oct 30 01:03:01.512490 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.512468    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istiod-ca-cert" pod "8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab" (UID: "8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:01.512600 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.512499    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-token\" (UniqueName: \"kubernetes.io/projected/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-token\") pod \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") "
Oct 30 01:03:01.512600 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.512497    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-data" pod "8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab" (UID: "8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:01.512600 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.512507    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-workload-certs" pod "8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab" (UID: "8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:01.512600 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.512585    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-workload-socket" pod "8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab" (UID: "8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:01.512781 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.512599    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-podinfo\") pod \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") "
Oct 30 01:03:01.512781 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.512673    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-credential-socket" pod "8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab" (UID: "8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:01.512781 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.512704    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-kube-api-access-frxnh\" (UniqueName: \"kubernetes.io/projected/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-kube-api-access-frxnh\") pod \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\" (UID: \"8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab\") "
Oct 30 01:03:01.513328 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.513305    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:03:01.513328 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.513327    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:03:01.513509 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.513337    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:03:01.513509 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.513347    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:03:01.513509 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.513355    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:03:01.514831 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.514803    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-podinfo" pod "8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab" (UID: "8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:03:01.514936 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.514914    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-kube-api-access-frxnh" pod "8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab" (UID: "8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab"). InnerVolumeSpecName "kube-api-access-frxnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:01.515065 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.515048    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-token" pod "8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab" (UID: "8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:01.515179 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.515112    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-envoy" pod "8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab" (UID: "8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:01.615215 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.615129    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:03:01.615215 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.615161    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:03:01.615215 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.615170    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:03:01.615215 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.615179    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frxnh\" (UniqueName: \"kubernetes.io/projected/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab-kube-api-access-frxnh\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:03:01.837655 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.837617    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"]
Oct 30 01:03:01.843937 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:01.843905    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-52b947-openshift-default-5984dd58f5-jszdp"]
Oct 30 01:03:03.794312 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:03.794265    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab" path="/var/lib/kubelet/pods/8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab/volumes"
Oct 30 01:03:04.763485 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.763448    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"]
Oct 30 01:03:04.764273 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.764247    4864 cpu_manager.go:414] "containerMap: removing container" podUID="8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab" containerName="istio-proxy"
Oct 30 01:03:04.764273 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.764268    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab" containerName="istio-proxy"
Oct 30 01:03:04.764432 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.764386    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab" containerName="istio-proxy"
Oct 30 01:03:04.769521 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.769493    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.786479 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.786454    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9a8d85-openshift-default-dockercfg-x8rls\""
Oct 30 01:03:04.801552 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.801524    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"]
Oct 30 01:03:04.864143 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.864107    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-envoy\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.864315 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.864185    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-data\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.864315 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.864265    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-podinfo\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.864315 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.864300    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-token\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.864460 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.864406    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-workload-socket\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.864460 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.864452    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istiod-ca-cert\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.864598 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.864576    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-credential-socket\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.864635 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.864624    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-workload-certs\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.864723 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.864708    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7v54\" (UniqueName: \"kubernetes.io/projected/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-kube-api-access-h7v54\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.966369 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.966332    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-credential-socket\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.966541 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.966377    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-workload-certs\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.966541 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.966447    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7v54\" (UniqueName: \"kubernetes.io/projected/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-kube-api-access-h7v54\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.966541 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.966487    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-envoy\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.966701 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.966558    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-data\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.966701 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.966683    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-podinfo\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.966794 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.966726    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-token\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.966850 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.966797    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-credential-socket\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.966911 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.966844    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-workload-certs\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.966911 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.966896    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-workload-socket\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.967009 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.966994    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istiod-ca-cert\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.967177 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.967154    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-workload-socket\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.967290 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.967270    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-data\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.967695 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.967672    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istiod-ca-cert\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.968685 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.968666    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-envoy\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.969359 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.969336    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-podinfo\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.987907 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.987875    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7v54\" (UniqueName: \"kubernetes.io/projected/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-kube-api-access-h7v54\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:04.988013 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:04.987881    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-token\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:05.083844 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:05.083755    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:05.415044 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:05.415018    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"]
Oct 30 01:03:05.416774 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:03:05.416744    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bbb6a1a_a1e6_44aa_aaf2_829c3acd2a7a.slice/crio-d1c8e06d67c0fe87c65161e8c0f3c83727de554a421c62295389bae5a91849db WatchSource:0}: Error finding container d1c8e06d67c0fe87c65161e8c0f3c83727de554a421c62295389bae5a91849db: Status 404 returned error can't find the container with id d1c8e06d67c0fe87c65161e8c0f3c83727de554a421c62295389bae5a91849db
Oct 30 01:03:05.419495 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:05.419459    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:03:05.419596 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:05.419531    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:03:05.419596 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:05.419557    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:03:05.506726 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:05.506685    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2" event={"ID":"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a","Type":"ContainerStarted","Data":"54e07b27c0dd6468aa5e74664f7edd08f6f74d75728b4ff17700b9b78bb466df"}
Oct 30 01:03:05.506882 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:05.506750    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2" event={"ID":"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a","Type":"ContainerStarted","Data":"d1c8e06d67c0fe87c65161e8c0f3c83727de554a421c62295389bae5a91849db"}
Oct 30 01:03:05.552521 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:05.552465    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2" podStartSLOduration=1.5524497259999999 podStartE2EDuration="1.552449726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:05.548251793 +0000 UTC m=+1297.886353717" watchObservedRunningTime="2026-10-30 01:03:05.552449726 +0000 UTC m=+1297.890551740"
Oct 30 01:03:06.089105 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:06.088767    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:06.090056 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:06.090025    4864 patch_prober.go:29] interesting pod/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 01:03:06.090364 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:06.090313    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2" podUID="7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 01:03:07.086460 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:07.086417    4864 patch_prober.go:29] interesting pod/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 01:03:07.086625 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:07.086488    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2" podUID="7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 01:03:08.085684 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:08.085644    4864 patch_prober.go:29] interesting pod/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 01:03:08.086320 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:08.086281    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2" podUID="7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 01:03:09.084723 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:09.084656    4864 patch_prober.go:29] interesting pod/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 01:03:09.085039 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:09.084744    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2" podUID="7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 01:03:10.160331 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:10.156822    4864 patch_prober.go:29] interesting pod/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 01:03:10.160331 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:10.156898    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2" podUID="7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 01:03:11.086193 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:11.086147    4864 patch_prober.go:29] interesting pod/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 01:03:11.086403 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:11.086236    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2" podUID="7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 01:03:12.088948 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:12.088917    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:12.089369 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:12.089312    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:12.089974 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:12.089953    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:03:19.043292 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:03:19.043255    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:03:20.038875 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.038839    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"]
Oct 30 01:03:20.039924 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.039896    4864 cpu_manager.go:414] "containerMap: removing container" podUID="8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab" containerName="istio-proxy"
Oct 30 01:03:20.039924 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.039923    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab" containerName="istio-proxy"
Oct 30 01:03:20.040152 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.040120    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eedbfd0-0fc7-43dc-a709-6a1dd6acd1ab" containerName="istio-proxy"
Oct 30 01:03:20.044404 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.044379    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.100812 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.100788    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1c78b8-openshift-default-dockercfg-x925j\""
Oct 30 01:03:20.105911 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.105879    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"]
Oct 30 01:03:20.135522 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.135490    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-data\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.135725 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.135549    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-podinfo\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.135725 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.135655    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db5bae2a-6063-4cea-95ed-36df20be18ef-istiod-ca-cert\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.135725 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.135697    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-envoy\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.135914 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.135760    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-workload-socket\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.135914 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.135827    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82f9r\" (UniqueName: \"kubernetes.io/projected/db5bae2a-6063-4cea-95ed-36df20be18ef-kube-api-access-82f9r\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.135914 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.135867    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-token\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.136101 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.135992    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-workload-certs\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.136101 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.136039    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-credential-socket\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.245247 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.245210    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-podinfo\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.245491 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.245472    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db5bae2a-6063-4cea-95ed-36df20be18ef-istiod-ca-cert\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.245589 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.245572    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-envoy\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.245676 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.245613    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-workload-socket\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.245676 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.245655    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82f9r\" (UniqueName: \"kubernetes.io/projected/db5bae2a-6063-4cea-95ed-36df20be18ef-kube-api-access-82f9r\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.245789 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.245726    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-token\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.245841 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.245817    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-workload-certs\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.245893 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.245866    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-credential-socket\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.246022 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.246005    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-data\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.246672 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.246641    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-data\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.264966 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.247127    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-workload-socket\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.265212 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.261263    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db5bae2a-6063-4cea-95ed-36df20be18ef-istiod-ca-cert\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.265324 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.263362    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-workload-certs\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.265419 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.263917    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-credential-socket\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.287101 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.272402    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-envoy\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.322414 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.320657    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-token\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.322414 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.321428    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82f9r\" (UniqueName: \"kubernetes.io/projected/db5bae2a-6063-4cea-95ed-36df20be18ef-kube-api-access-82f9r\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.346093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.342680    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-podinfo\") pod \"e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") " pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.410593 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.410563    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:20.803526 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.803497    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"]
Oct 30 01:03:20.805120 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:03:20.805058    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb5bae2a_6063_4cea_95ed_36df20be18ef.slice/crio-9cc9011c469aac74f881808baa1ea77a7f7792ee987be00bfa1518e3620952e8 WatchSource:0}: Error finding container 9cc9011c469aac74f881808baa1ea77a7f7792ee987be00bfa1518e3620952e8: Status 404 returned error can't find the container with id 9cc9011c469aac74f881808baa1ea77a7f7792ee987be00bfa1518e3620952e8
Oct 30 01:03:20.807691 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.807657    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:03:20.807776 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.807723    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:03:20.807776 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:20.807752    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:03:21.810622 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:21.810581    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh" event={"ID":"db5bae2a-6063-4cea-95ed-36df20be18ef","Type":"ContainerStarted","Data":"f3a8dcea9ac3e886b0eb439175a271955563c53363326b2d0c2c6f3ef4a82bb6"}
Oct 30 01:03:21.811012 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:21.810678    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh" event={"ID":"db5bae2a-6063-4cea-95ed-36df20be18ef","Type":"ContainerStarted","Data":"9cc9011c469aac74f881808baa1ea77a7f7792ee987be00bfa1518e3620952e8"}
Oct 30 01:03:21.912096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:21.897644    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh" podStartSLOduration=1.897623139 podStartE2EDuration="1.897623139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:21.871593596 +0000 UTC m=+1314.209695519" watchObservedRunningTime="2026-10-30 01:03:21.897623139 +0000 UTC m=+1314.235725051"
Oct 30 01:03:22.411137 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:22.411107    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:22.416175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:22.416148    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:22.817944 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:22.817913    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:22.819027 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:22.819008    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:03:28.191311 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:28.191268    4864 patch_prober.go:29] interesting pod/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:03:28.191783 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:28.191359    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt" podUID="e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:03:29.884393 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:29.884358    4864 generic.go:356] "Generic (PLEG): container finished" podID="e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" containerID="c6d4f8300c978be5d76ad8699a85f890b7f3f23b49a3f3cbad9965ba4a19d487" exitCode=0
Oct 30 01:03:29.884833 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:29.884432    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt" event={"ID":"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b","Type":"ContainerDied","Data":"c6d4f8300c978be5d76ad8699a85f890b7f3f23b49a3f3cbad9965ba4a19d487"}
Oct 30 01:03:29.884938 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:29.884924    4864 scope.go:122] "RemoveContainer" containerID="c6d4f8300c978be5d76ad8699a85f890b7f3f23b49a3f3cbad9965ba4a19d487"
Oct 30 01:03:29.886576 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:29.886548    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:03:29.886641 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:29.886620    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:03:29.886676 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:29.886651    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:03:30.894644 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:30.894612    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt" event={"ID":"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b","Type":"ContainerStarted","Data":"74d8065112360c40433132ec829ea281a81c0ddad460bc2d28b9b54f7a76242b"}
Oct 30 01:03:30.987996 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:30.987962    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:03:30.988383 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:30.988356    4864 patch_prober.go:29] interesting pod/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 01:03:30.988475 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:30.988416    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt" podUID="e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 01:03:31.982577 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:31.982532    4864 patch_prober.go:29] interesting pod/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 01:03:31.983154 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:31.982600    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt" podUID="e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 01:03:32.975512 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:32.975470    4864 patch_prober.go:29] interesting pod/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 01:03:32.976995 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:32.976956    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt" podUID="e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 01:03:33.383994 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:33.383909    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5"]
Oct 30 01:03:33.389317 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:33.389295    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5"
Oct 30 01:03:33.402168 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:33.402146    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-645126-serving-cert\""
Oct 30 01:03:33.419652 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:33.419620    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5"]
Oct 30 01:03:33.506438 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:33.506402    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe1b890e-9648-4210-aa76-ef90b4ad983c-maas-api-tls\") pod \"maas-api-e2e-api-a-645126-58cb667c69-9bpf5\" (UID: \"fe1b890e-9648-4210-aa76-ef90b4ad983c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5"
Oct 30 01:03:33.506627 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:33.506610    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe1b890e-9648-4210-aa76-ef90b4ad983c-metrics-certs\") pod \"maas-api-e2e-api-a-645126-58cb667c69-9bpf5\" (UID: \"fe1b890e-9648-4210-aa76-ef90b4ad983c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5"
Oct 30 01:03:33.506747 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:33.506725    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmxlc\" (UniqueName: \"kubernetes.io/projected/fe1b890e-9648-4210-aa76-ef90b4ad983c-kube-api-access-qmxlc\") pod \"maas-api-e2e-api-a-645126-58cb667c69-9bpf5\" (UID: \"fe1b890e-9648-4210-aa76-ef90b4ad983c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5"
Oct 30 01:03:33.506848 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:33.506838    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1b890e-9648-4210-aa76-ef90b4ad983c-openshift-service-ca\") pod \"maas-api-e2e-api-a-645126-58cb667c69-9bpf5\" (UID: \"fe1b890e-9648-4210-aa76-ef90b4ad983c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5"
Oct 30 01:03:33.608322 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:33.608289    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe1b890e-9648-4210-aa76-ef90b4ad983c-metrics-certs\") pod \"maas-api-e2e-api-a-645126-58cb667c69-9bpf5\" (UID: \"fe1b890e-9648-4210-aa76-ef90b4ad983c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5"
Oct 30 01:03:33.608503 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:33.608332    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmxlc\" (UniqueName: \"kubernetes.io/projected/fe1b890e-9648-4210-aa76-ef90b4ad983c-kube-api-access-qmxlc\") pod \"maas-api-e2e-api-a-645126-58cb667c69-9bpf5\" (UID: \"fe1b890e-9648-4210-aa76-ef90b4ad983c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5"
Oct 30 01:03:33.608503 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:33.608377    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1b890e-9648-4210-aa76-ef90b4ad983c-openshift-service-ca\") pod \"maas-api-e2e-api-a-645126-58cb667c69-9bpf5\" (UID: \"fe1b890e-9648-4210-aa76-ef90b4ad983c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5"
Oct 30 01:03:33.608600 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:33.608510    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe1b890e-9648-4210-aa76-ef90b4ad983c-maas-api-tls\") pod \"maas-api-e2e-api-a-645126-58cb667c69-9bpf5\" (UID: \"fe1b890e-9648-4210-aa76-ef90b4ad983c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5"
Oct 30 01:03:33.609237 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:33.609208    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1b890e-9648-4210-aa76-ef90b4ad983c-openshift-service-ca\") pod \"maas-api-e2e-api-a-645126-58cb667c69-9bpf5\" (UID: \"fe1b890e-9648-4210-aa76-ef90b4ad983c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5"
Oct 30 01:03:33.610934 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:33.610906    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe1b890e-9648-4210-aa76-ef90b4ad983c-maas-api-tls\") pod \"maas-api-e2e-api-a-645126-58cb667c69-9bpf5\" (UID: \"fe1b890e-9648-4210-aa76-ef90b4ad983c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5"
Oct 30 01:03:33.610934 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:33.610928    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe1b890e-9648-4210-aa76-ef90b4ad983c-metrics-certs\") pod \"maas-api-e2e-api-a-645126-58cb667c69-9bpf5\" (UID: \"fe1b890e-9648-4210-aa76-ef90b4ad983c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5"
Oct 30 01:03:33.630354 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:33.630329    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmxlc\" (UniqueName: \"kubernetes.io/projected/fe1b890e-9648-4210-aa76-ef90b4ad983c-kube-api-access-qmxlc\") pod \"maas-api-e2e-api-a-645126-58cb667c69-9bpf5\" (UID: \"fe1b890e-9648-4210-aa76-ef90b4ad983c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5"
Oct 30 01:03:33.707260 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:33.704722    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5"
Oct 30 01:03:33.981784 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:33.981228    4864 patch_prober.go:29] interesting pod/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 01:03:33.981784 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:33.981300    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt" podUID="e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 01:03:34.086208 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:34.086170    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5"]
Oct 30 01:03:34.086344 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:03:34.086235    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe1b890e_9648_4210_aa76_ef90b4ad983c.slice/crio-d2eb9524c3a1c39267865958671d93f9a88885e06555854e49598e12697c0812 WatchSource:0}: Error finding container d2eb9524c3a1c39267865958671d93f9a88885e06555854e49598e12697c0812: Status 404 returned error can't find the container with id d2eb9524c3a1c39267865958671d93f9a88885e06555854e49598e12697c0812
Oct 30 01:03:34.931043 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:34.931011    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5" event={"ID":"fe1b890e-9648-4210-aa76-ef90b4ad983c","Type":"ContainerStarted","Data":"4a4a50918d8b3403ef70611582ff40fa0f860078a49f479c820c180637547b04"}
Oct 30 01:03:34.931444 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:34.931116    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5" event={"ID":"fe1b890e-9648-4210-aa76-ef90b4ad983c","Type":"ContainerStarted","Data":"d2eb9524c3a1c39267865958671d93f9a88885e06555854e49598e12697c0812"}
Oct 30 01:03:34.932032 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:34.932013    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5"
Oct 30 01:03:34.979096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:34.976607    4864 patch_prober.go:29] interesting pod/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 01:03:34.979096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:34.976681    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt" podUID="e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 01:03:34.984628 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:34.984511    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5" podStartSLOduration=1.635730868 podStartE2EDuration="1.984488773s" totalImagesPullingTime="348.757905ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:34.973689334 +0000 UTC m=+1327.311791247" watchObservedRunningTime="2026-10-30 01:03:34.984488773 +0000 UTC m=+1327.322590685"
Oct 30 01:03:35.976122 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:35.976029    4864 patch_prober.go:29] interesting pod/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 01:03:35.976122 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:35.976102    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt" podUID="e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 01:03:36.978584 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:36.978550    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:03:36.978970 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:36.978932    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:03:36.979547 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:36.979525    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:03:42.009357 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:42.009326    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5"
Oct 30 01:03:49.041189 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:03:49.041150    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:03:52.299477 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:52.299435    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7294-fe5171"]
Oct 30 01:03:52.303042 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:52.303016    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7294-fe5171"
Oct 30 01:03:52.317440 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:52.317414    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4z6vl\""
Oct 30 01:03:52.321370 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:52.321343    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7294-fe5171"]
Oct 30 01:03:52.460927 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:52.460890    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbh99\" (UniqueName: \"kubernetes.io/projected/b8867b89-e79f-4fab-a88c-a5d4c154f4bf-kube-api-access-gbh99\") pod \"test-curl-7294-fe5171\" (UID: \"b8867b89-e79f-4fab-a88c-a5d4c154f4bf\") " pod="opendatahub/test-curl-7294-fe5171"
Oct 30 01:03:52.562927 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:52.562842    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbh99\" (UniqueName: \"kubernetes.io/projected/b8867b89-e79f-4fab-a88c-a5d4c154f4bf-kube-api-access-gbh99\") pod \"test-curl-7294-fe5171\" (UID: \"b8867b89-e79f-4fab-a88c-a5d4c154f4bf\") " pod="opendatahub/test-curl-7294-fe5171"
Oct 30 01:03:52.584003 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:52.583976    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbh99\" (UniqueName: \"kubernetes.io/projected/b8867b89-e79f-4fab-a88c-a5d4c154f4bf-kube-api-access-gbh99\") pod \"test-curl-7294-fe5171\" (UID: \"b8867b89-e79f-4fab-a88c-a5d4c154f4bf\") " pod="opendatahub/test-curl-7294-fe5171"
Oct 30 01:03:52.615789 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:52.615752    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7294-fe5171"
Oct 30 01:03:52.923454 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:52.923190    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7294-fe5171"]
Oct 30 01:03:52.926016 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:03:52.925984    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8867b89_e79f_4fab_a88c_a5d4c154f4bf.slice/crio-3d948b306d90fccdc1866bc50034a5b18c8d00d7ca50b308af1429a682044efe WatchSource:0}: Error finding container 3d948b306d90fccdc1866bc50034a5b18c8d00d7ca50b308af1429a682044efe: Status 404 returned error can't find the container with id 3d948b306d90fccdc1866bc50034a5b18c8d00d7ca50b308af1429a682044efe
Oct 30 01:03:53.112484 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:53.112440    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7294-fe5171" event={"ID":"b8867b89-e79f-4fab-a88c-a5d4c154f4bf","Type":"ContainerStarted","Data":"3d948b306d90fccdc1866bc50034a5b18c8d00d7ca50b308af1429a682044efe"}
Oct 30 01:03:56.139514 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:56.139473    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7294-fe5171" event={"ID":"b8867b89-e79f-4fab-a88c-a5d4c154f4bf","Type":"ContainerStarted","Data":"afec9cc963599a3a05e9bb930860ef09997a90fbac880e4604c3ed7a51f29357"}
Oct 30 01:03:56.175551 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:56.175493    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7294-fe5171" podStartSLOduration=1.760132407 podStartE2EDuration="4.175479429s" totalImagesPullingTime="2.415347022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:56.173299222 +0000 UTC m=+1348.511401131" watchObservedRunningTime="2026-10-30 01:03:56.175479429 +0000 UTC m=+1348.513581338"
Oct 30 01:03:57.228759 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:57.228722    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7294-fe5171"]
Oct 30 01:03:57.234223 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:57.234183    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7294-fe5171"]
Oct 30 01:03:58.154508 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:03:58.154455    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7294-fe5171" podUID="b8867b89-e79f-4fab-a88c-a5d4c154f4bf" containerName="test-curl-7294-fe5171" containerID="cri-o://afec9cc963599a3a05e9bb930860ef09997a90fbac880e4604c3ed7a51f29357" gracePeriod=2
Oct 30 01:04:00.691772 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:00.691746    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7294-fe5171"
Oct 30 01:04:00.747992 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:00.747898    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8867b89-e79f-4fab-a88c-a5d4c154f4bf-kube-api-access-gbh99\" (UniqueName: \"kubernetes.io/projected/b8867b89-e79f-4fab-a88c-a5d4c154f4bf-kube-api-access-gbh99\") pod \"b8867b89-e79f-4fab-a88c-a5d4c154f4bf\" (UID: \"b8867b89-e79f-4fab-a88c-a5d4c154f4bf\") "
Oct 30 01:04:00.750024 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:00.749993    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8867b89-e79f-4fab-a88c-a5d4c154f4bf-kube-api-access-gbh99" pod "b8867b89-e79f-4fab-a88c-a5d4c154f4bf" (UID: "b8867b89-e79f-4fab-a88c-a5d4c154f4bf"). InnerVolumeSpecName "kube-api-access-gbh99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:00.852143 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:00.850478    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbh99\" (UniqueName: \"kubernetes.io/projected/b8867b89-e79f-4fab-a88c-a5d4c154f4bf-kube-api-access-gbh99\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:04:01.192828 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:01.192733    4864 generic.go:356] "Generic (PLEG): container finished" podID="b8867b89-e79f-4fab-a88c-a5d4c154f4bf" containerID="afec9cc963599a3a05e9bb930860ef09997a90fbac880e4604c3ed7a51f29357" exitCode=137
Oct 30 01:04:01.192979 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:01.192931    4864 scope.go:122] "RemoveContainer" containerID="afec9cc963599a3a05e9bb930860ef09997a90fbac880e4604c3ed7a51f29357"
Oct 30 01:04:01.193088 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:01.193061    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7294-fe5171"
Oct 30 01:04:01.204643 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:01.204625    4864 scope.go:122] "RemoveContainer" containerID="afec9cc963599a3a05e9bb930860ef09997a90fbac880e4604c3ed7a51f29357"
Oct 30 01:04:01.204923 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:04:01.204903    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afec9cc963599a3a05e9bb930860ef09997a90fbac880e4604c3ed7a51f29357\": container with ID starting with afec9cc963599a3a05e9bb930860ef09997a90fbac880e4604c3ed7a51f29357 not found: ID does not exist" containerID="afec9cc963599a3a05e9bb930860ef09997a90fbac880e4604c3ed7a51f29357"
Oct 30 01:04:01.204971 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:01.204932    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afec9cc963599a3a05e9bb930860ef09997a90fbac880e4604c3ed7a51f29357"} err="failed to get container status \"afec9cc963599a3a05e9bb930860ef09997a90fbac880e4604c3ed7a51f29357\": rpc error: code = NotFound desc = could not find container \"afec9cc963599a3a05e9bb930860ef09997a90fbac880e4604c3ed7a51f29357\": container with ID starting with afec9cc963599a3a05e9bb930860ef09997a90fbac880e4604c3ed7a51f29357 not found: ID does not exist"
Oct 30 01:04:01.214699 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:04:01.214667    4864 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7294-fe5171\" is forbidden: User \"system:node:ip-10-0-134-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-210.ec2.internal' and this object" podUID="b8867b89-e79f-4fab-a88c-a5d4c154f4bf" pod="opendatahub/test-curl-7294-fe5171"
Oct 30 01:04:01.794431 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:01.794395    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8867b89-e79f-4fab-a88c-a5d4c154f4bf" path="/var/lib/kubelet/pods/b8867b89-e79f-4fab-a88c-a5d4c154f4bf/volumes"
Oct 30 01:04:08.859273 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:08.859232    4864 patch_prober.go:29] interesting pod/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:04:08.859694 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:08.859327    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh" podUID="db5bae2a-6063-4cea-95ed-36df20be18ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:10.282206 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:10.282161    4864 generic.go:356] "Generic (PLEG): container finished" podID="db5bae2a-6063-4cea-95ed-36df20be18ef" containerID="f3a8dcea9ac3e886b0eb439175a271955563c53363326b2d0c2c6f3ef4a82bb6" exitCode=0
Oct 30 01:04:10.282842 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:10.282187    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh" event={"ID":"db5bae2a-6063-4cea-95ed-36df20be18ef","Type":"ContainerDied","Data":"f3a8dcea9ac3e886b0eb439175a271955563c53363326b2d0c2c6f3ef4a82bb6"}
Oct 30 01:04:10.282907 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:10.282845    4864 scope.go:122] "RemoveContainer" containerID="f3a8dcea9ac3e886b0eb439175a271955563c53363326b2d0c2c6f3ef4a82bb6"
Oct 30 01:04:10.285602 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:10.285570    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:04:10.285736 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:10.285661    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:04:10.285736 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:10.285711    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:04:11.292175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:11.292131    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh" event={"ID":"db5bae2a-6063-4cea-95ed-36df20be18ef","Type":"ContainerStarted","Data":"5fcb6b745a504bf0ed5fb8a1662d2c4474c5601d63060ca28eca9d54ef5e1142"}
Oct 30 01:04:11.411989 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:11.411952    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:04:11.412248 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:11.412225    4864 patch_prober.go:29] interesting pod/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 01:04:11.412335 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:11.412276    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh" podUID="db5bae2a-6063-4cea-95ed-36df20be18ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 01:04:12.411942 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:12.411906    4864 patch_prober.go:29] interesting pod/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 01:04:12.412410 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:12.411974    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh" podUID="db5bae2a-6063-4cea-95ed-36df20be18ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 01:04:13.411950 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:13.411913    4864 patch_prober.go:29] interesting pod/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 01:04:13.412370 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:13.411977    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh" podUID="db5bae2a-6063-4cea-95ed-36df20be18ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 01:04:14.411619 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:14.411586    4864 patch_prober.go:29] interesting pod/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 01:04:14.411784 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:14.411648    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh" podUID="db5bae2a-6063-4cea-95ed-36df20be18ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 01:04:15.411480 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:15.411442    4864 patch_prober.go:29] interesting pod/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 01:04:15.411909 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:15.411508    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh" podUID="db5bae2a-6063-4cea-95ed-36df20be18ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 01:04:16.467193 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:16.467153    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:04:16.467770 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:16.467709    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:04:16.468754 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:16.468731    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:04:19.044563 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:04:19.044528    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:04:34.843753 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:34.843715    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"]
Oct 30 01:04:34.844519 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:34.843961    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj" podUID="254d9154-d8b0-4058-9ebf-8963632489c7" containerName="maas-api" containerID="cri-o://378582ded3694f4fc3eb8aab61f28401e3e92151da3036308cb132810c7f52a9" gracePeriod=30
Oct 30 01:04:35.183568 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:35.183537    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"
Oct 30 01:04:35.278681 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:35.278644    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/254d9154-d8b0-4058-9ebf-8963632489c7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254d9154-d8b0-4058-9ebf-8963632489c7-maas-api-tls\") pod \"254d9154-d8b0-4058-9ebf-8963632489c7\" (UID: \"254d9154-d8b0-4058-9ebf-8963632489c7\") "
Oct 30 01:04:35.278887 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:35.278727    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/254d9154-d8b0-4058-9ebf-8963632489c7-kube-api-access-k5rbk\" (UniqueName: \"kubernetes.io/projected/254d9154-d8b0-4058-9ebf-8963632489c7-kube-api-access-k5rbk\") pod \"254d9154-d8b0-4058-9ebf-8963632489c7\" (UID: \"254d9154-d8b0-4058-9ebf-8963632489c7\") "
Oct 30 01:04:35.278887 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:35.278835    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/254d9154-d8b0-4058-9ebf-8963632489c7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254d9154-d8b0-4058-9ebf-8963632489c7-openshift-service-ca\") pod \"254d9154-d8b0-4058-9ebf-8963632489c7\" (UID: \"254d9154-d8b0-4058-9ebf-8963632489c7\") "
Oct 30 01:04:35.279002 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:35.278909    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/254d9154-d8b0-4058-9ebf-8963632489c7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/254d9154-d8b0-4058-9ebf-8963632489c7-metrics-certs\") pod \"254d9154-d8b0-4058-9ebf-8963632489c7\" (UID: \"254d9154-d8b0-4058-9ebf-8963632489c7\") "
Oct 30 01:04:35.279528 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:35.279478    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/254d9154-d8b0-4058-9ebf-8963632489c7-openshift-service-ca" pod "254d9154-d8b0-4058-9ebf-8963632489c7" (UID: "254d9154-d8b0-4058-9ebf-8963632489c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:35.279748 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:35.279727    4864 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254d9154-d8b0-4058-9ebf-8963632489c7-openshift-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:04:35.292357 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:35.284150    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/254d9154-d8b0-4058-9ebf-8963632489c7-maas-api-tls" pod "254d9154-d8b0-4058-9ebf-8963632489c7" (UID: "254d9154-d8b0-4058-9ebf-8963632489c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:04:35.292357 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:35.289784    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/254d9154-d8b0-4058-9ebf-8963632489c7-metrics-certs" pod "254d9154-d8b0-4058-9ebf-8963632489c7" (UID: "254d9154-d8b0-4058-9ebf-8963632489c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:04:35.292357 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:35.291479    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254d9154-d8b0-4058-9ebf-8963632489c7-kube-api-access-k5rbk" pod "254d9154-d8b0-4058-9ebf-8963632489c7" (UID: "254d9154-d8b0-4058-9ebf-8963632489c7"). InnerVolumeSpecName "kube-api-access-k5rbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:35.381785 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:35.381747    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254d9154-d8b0-4058-9ebf-8963632489c7-maas-api-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:04:35.381785 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:35.381785    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5rbk\" (UniqueName: \"kubernetes.io/projected/254d9154-d8b0-4058-9ebf-8963632489c7-kube-api-access-k5rbk\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:04:35.382039 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:35.381797    4864 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254d9154-d8b0-4058-9ebf-8963632489c7-metrics-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:04:35.496837 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:35.496800    4864 generic.go:356] "Generic (PLEG): container finished" podID="254d9154-d8b0-4058-9ebf-8963632489c7" containerID="378582ded3694f4fc3eb8aab61f28401e3e92151da3036308cb132810c7f52a9" exitCode=0
Oct 30 01:04:35.497062 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:35.496850    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"
Oct 30 01:04:35.497062 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:35.496864    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj" event={"ID":"254d9154-d8b0-4058-9ebf-8963632489c7","Type":"ContainerDied","Data":"378582ded3694f4fc3eb8aab61f28401e3e92151da3036308cb132810c7f52a9"}
Oct 30 01:04:35.497062 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:35.496914    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj" event={"ID":"254d9154-d8b0-4058-9ebf-8963632489c7","Type":"ContainerDied","Data":"3d9d02216997a5cd4fa9d5df5b80e6b43ecfbbdf0ac52b1e4be739c43d78b5b3"}
Oct 30 01:04:35.497062 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:35.496942    4864 scope.go:122] "RemoveContainer" containerID="378582ded3694f4fc3eb8aab61f28401e3e92151da3036308cb132810c7f52a9"
Oct 30 01:04:35.513024 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:35.513003    4864 scope.go:122] "RemoveContainer" containerID="378582ded3694f4fc3eb8aab61f28401e3e92151da3036308cb132810c7f52a9"
Oct 30 01:04:35.513383 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:04:35.513361    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"378582ded3694f4fc3eb8aab61f28401e3e92151da3036308cb132810c7f52a9\": container with ID starting with 378582ded3694f4fc3eb8aab61f28401e3e92151da3036308cb132810c7f52a9 not found: ID does not exist" containerID="378582ded3694f4fc3eb8aab61f28401e3e92151da3036308cb132810c7f52a9"
Oct 30 01:04:35.513463 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:35.513392    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"378582ded3694f4fc3eb8aab61f28401e3e92151da3036308cb132810c7f52a9"} err="failed to get container status \"378582ded3694f4fc3eb8aab61f28401e3e92151da3036308cb132810c7f52a9\": rpc error: code = NotFound desc = could not find container \"378582ded3694f4fc3eb8aab61f28401e3e92151da3036308cb132810c7f52a9\": container with ID starting with 378582ded3694f4fc3eb8aab61f28401e3e92151da3036308cb132810c7f52a9 not found: ID does not exist"
Oct 30 01:04:35.569564 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:35.569532    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"]
Oct 30 01:04:35.581678 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:35.581645    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9a8d85-6586784bc9-zjstj"]
Oct 30 01:04:35.800665 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:35.800565    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="254d9154-d8b0-4058-9ebf-8963632489c7" path="/var/lib/kubelet/pods/254d9154-d8b0-4058-9ebf-8963632489c7/volumes"
Oct 30 01:04:39.299810 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:39.299768    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv"]
Oct 30 01:04:39.300641 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:39.300617    4864 cpu_manager.go:414] "containerMap: removing container" podUID="b8867b89-e79f-4fab-a88c-a5d4c154f4bf" containerName="test-curl-7294-fe5171"
Oct 30 01:04:39.300766 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:39.300641    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8867b89-e79f-4fab-a88c-a5d4c154f4bf" containerName="test-curl-7294-fe5171"
Oct 30 01:04:39.300766 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:39.300658    4864 cpu_manager.go:414] "containerMap: removing container" podUID="254d9154-d8b0-4058-9ebf-8963632489c7" containerName="maas-api"
Oct 30 01:04:39.300766 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:39.300665    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254d9154-d8b0-4058-9ebf-8963632489c7" containerName="maas-api"
Oct 30 01:04:39.300917 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:39.300851    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8867b89-e79f-4fab-a88c-a5d4c154f4bf" containerName="test-curl-7294-fe5171"
Oct 30 01:04:39.300917 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:39.300867    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="254d9154-d8b0-4058-9ebf-8963632489c7" containerName="maas-api"
Oct 30 01:04:39.305963 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:39.305929    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv"
Oct 30 01:04:39.322887 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:39.322860    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d4622a\""
Oct 30 01:04:39.326621 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:39.326595    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv"]
Oct 30 01:04:39.338659 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:39.338635    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqnvf\" (UniqueName: \"kubernetes.io/projected/bea59237-06d5-4067-8057-1b87d5dd7f4c-kube-api-access-wqnvf\") pod \"payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv\" (UID: \"bea59237-06d5-4067-8057-1b87d5dd7f4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv"
Oct 30 01:04:39.338782 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:39.338766    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea59237-06d5-4067-8057-1b87d5dd7f4c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv\" (UID: \"bea59237-06d5-4067-8057-1b87d5dd7f4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv"
Oct 30 01:04:39.405109 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:39.405048    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d4622a-dockercfg-5xjlg\""
Oct 30 01:04:39.446095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:39.442559    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea59237-06d5-4067-8057-1b87d5dd7f4c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv\" (UID: \"bea59237-06d5-4067-8057-1b87d5dd7f4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv"
Oct 30 01:04:39.446095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:39.442857    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqnvf\" (UniqueName: \"kubernetes.io/projected/bea59237-06d5-4067-8057-1b87d5dd7f4c-kube-api-access-wqnvf\") pod \"payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv\" (UID: \"bea59237-06d5-4067-8057-1b87d5dd7f4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv"
Oct 30 01:04:39.446343 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:39.446277    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea59237-06d5-4067-8057-1b87d5dd7f4c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv\" (UID: \"bea59237-06d5-4067-8057-1b87d5dd7f4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv"
Oct 30 01:04:39.527091 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:39.526198    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqnvf\" (UniqueName: \"kubernetes.io/projected/bea59237-06d5-4067-8057-1b87d5dd7f4c-kube-api-access-wqnvf\") pod \"payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv\" (UID: \"bea59237-06d5-4067-8057-1b87d5dd7f4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv"
Oct 30 01:04:39.620179 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:39.620064    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv"
Oct 30 01:04:39.931597 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:04:39.930885    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea59237_06d5_4067_8057_1b87d5dd7f4c.slice/crio-ac96e3a083a0c200c09ebd7a7cf09905e5c2aae685d9b5d139141e6ab747271b WatchSource:0}: Error finding container ac96e3a083a0c200c09ebd7a7cf09905e5c2aae685d9b5d139141e6ab747271b: Status 404 returned error can't find the container with id ac96e3a083a0c200c09ebd7a7cf09905e5c2aae685d9b5d139141e6ab747271b
Oct 30 01:04:39.931597 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:39.931312    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv"]
Oct 30 01:04:40.540578 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:40.540545    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv" event={"ID":"bea59237-06d5-4067-8057-1b87d5dd7f4c","Type":"ContainerStarted","Data":"4b8e51b4652f49870959a62a1d9be29fb349b40d7aab3115df17ab629f458665"}
Oct 30 01:04:40.541018 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:40.540602    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv" event={"ID":"bea59237-06d5-4067-8057-1b87d5dd7f4c","Type":"ContainerStarted","Data":"ac96e3a083a0c200c09ebd7a7cf09905e5c2aae685d9b5d139141e6ab747271b"}
Oct 30 01:04:40.541018 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:40.540633    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv"
Oct 30 01:04:40.591576 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:40.591514    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv" podStartSLOduration=1.591497336 podStartE2EDuration="1.591497336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:40.576996699 +0000 UTC m=+1392.915098600" watchObservedRunningTime="2026-10-30 01:04:40.591497336 +0000 UTC m=+1392.929599265"
Oct 30 01:04:42.649491 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:42.649452    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"]
Oct 30 01:04:42.649903 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:42.649699    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2" podUID="7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a" containerName="istio-proxy" containerID="cri-o://54e07b27c0dd6468aa5e74664f7edd08f6f74d75728b4ff17700b9b78bb466df" gracePeriod=30
Oct 30 01:04:47.921294 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:47.921219    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:04:48.033811 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.033773    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-podinfo\") pod \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") "
Oct 30 01:04:48.034016 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.033829    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-kube-api-access-h7v54\" (UniqueName: \"kubernetes.io/projected/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-kube-api-access-h7v54\") pod \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") "
Oct 30 01:04:48.034016 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.033870    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-workload-socket\") pod \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") "
Oct 30 01:04:48.034016 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.033931    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istiod-ca-cert\") pod \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") "
Oct 30 01:04:48.034016 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.033976    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-data\") pod \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") "
Oct 30 01:04:48.034246 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.034139    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-credential-socket\") pod \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") "
Oct 30 01:04:48.034334 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.034309    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-envoy\") pod \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") "
Oct 30 01:04:48.034471 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.034352    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-token\" (UniqueName: \"kubernetes.io/projected/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-token\") pod \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") "
Oct 30 01:04:48.034471 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.034429    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-workload-certs\") pod \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\" (UID: \"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a\") "
Oct 30 01:04:48.034575 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.034553    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-workload-socket" pod "7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a" (UID: "7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:48.034790 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.034767    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-credential-socket" pod "7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a" (UID: "7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:48.035001 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.034979    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-data" pod "7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a" (UID: "7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:48.035122 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.035062    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istiod-ca-cert" pod "7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a" (UID: "7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:48.035253 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.035236    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.035334 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.035258    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.035334 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.035273    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.035334 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.035286    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.035578 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.035557    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-workload-certs" pod "7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a" (UID: "7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:48.037017 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.036977    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-kube-api-access-h7v54" pod "7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a" (UID: "7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a"). InnerVolumeSpecName "kube-api-access-h7v54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:48.037636 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.037607    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-envoy" pod "7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a" (UID: "7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:48.037749 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.037732    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-token" pod "7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a" (UID: "7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:48.037920 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.037904    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-podinfo" pod "7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a" (UID: "7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:04:48.136927 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.136892    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.136927 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.136927    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.137176 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.136942    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.137176 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.136954    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.137176 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.136969    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7v54\" (UniqueName: \"kubernetes.io/projected/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a-kube-api-access-h7v54\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.598363 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.598323    4864 generic.go:356] "Generic (PLEG): container finished" podID="7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a" containerID="54e07b27c0dd6468aa5e74664f7edd08f6f74d75728b4ff17700b9b78bb466df" exitCode=0
Oct 30 01:04:48.598363 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.598373    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"
Oct 30 01:04:48.598605 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.598419    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2" event={"ID":"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a","Type":"ContainerDied","Data":"54e07b27c0dd6468aa5e74664f7edd08f6f74d75728b4ff17700b9b78bb466df"}
Oct 30 01:04:48.598605 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.598492    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2" event={"ID":"7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a","Type":"ContainerDied","Data":"d1c8e06d67c0fe87c65161e8c0f3c83727de554a421c62295389bae5a91849db"}
Oct 30 01:04:48.598605 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.598525    4864 scope.go:122] "RemoveContainer" containerID="54e07b27c0dd6468aa5e74664f7edd08f6f74d75728b4ff17700b9b78bb466df"
Oct 30 01:04:48.614129 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.614098    4864 scope.go:122] "RemoveContainer" containerID="54e07b27c0dd6468aa5e74664f7edd08f6f74d75728b4ff17700b9b78bb466df"
Oct 30 01:04:48.614487 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:04:48.614467    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54e07b27c0dd6468aa5e74664f7edd08f6f74d75728b4ff17700b9b78bb466df\": container with ID starting with 54e07b27c0dd6468aa5e74664f7edd08f6f74d75728b4ff17700b9b78bb466df not found: ID does not exist" containerID="54e07b27c0dd6468aa5e74664f7edd08f6f74d75728b4ff17700b9b78bb466df"
Oct 30 01:04:48.614553 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.614496    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e07b27c0dd6468aa5e74664f7edd08f6f74d75728b4ff17700b9b78bb466df"} err="failed to get container status \"54e07b27c0dd6468aa5e74664f7edd08f6f74d75728b4ff17700b9b78bb466df\": rpc error: code = NotFound desc = could not find container \"54e07b27c0dd6468aa5e74664f7edd08f6f74d75728b4ff17700b9b78bb466df\": container with ID starting with 54e07b27c0dd6468aa5e74664f7edd08f6f74d75728b4ff17700b9b78bb466df not found: ID does not exist"
Oct 30 01:04:48.657675 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.657640    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"]
Oct 30 01:04:48.668699 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:48.668665    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-5mcl2"]
Oct 30 01:04:49.041544 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:04:49.041508    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:04:49.795439 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:49.795402    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a" path="/var/lib/kubelet/pods/7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a/volumes"
Oct 30 01:04:51.551318 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:04:51.551289    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv"
Oct 30 01:05:00.825407 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:00.825370    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"]
Oct 30 01:05:00.825905 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:00.825657    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt" podUID="e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" containerName="istio-proxy" containerID="cri-o://74d8065112360c40433132ec829ea281a81c0ddad460bc2d28b9b54f7a76242b" gracePeriod=30
Oct 30 01:05:03.698307 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.698273    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"]
Oct 30 01:05:03.698939 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.698922    4864 cpu_manager.go:414] "containerMap: removing container" podUID="7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a" containerName="istio-proxy"
Oct 30 01:05:03.699022 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.698940    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a" containerName="istio-proxy"
Oct 30 01:05:03.699126 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.699113    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bbb6a1a-a1e6-44aa-aaf2-829c3acd2a7a" containerName="istio-proxy"
Oct 30 01:05:03.703110 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.703060    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.717609 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.717574    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-76f654-openshift-default-dockercfg-p6m9l\""
Oct 30 01:05:03.732411 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.732383    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"]
Oct 30 01:05:03.792882 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.792847    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-podinfo\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.793046 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.792965    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-data\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.793123 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.793053    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-workload-certs\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.793171 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.793135    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istiod-ca-cert\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.793229 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.793205    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5v2k\" (UniqueName: \"kubernetes.io/projected/5e60eb51-e478-4cdc-b604-98e106f6bb4f-kube-api-access-n5v2k\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.793264 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.793249    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-credential-socket\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.793434 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.793422    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-workload-socket\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.793480 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.793469    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-token\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.793537 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.793526    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-envoy\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.895522 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.895488    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-workload-socket\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.895702 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.895545    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-token\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.895702 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.895576    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-envoy\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.895702 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.895598    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-podinfo\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.895702 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.895664    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-data\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.895911 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.895766    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-workload-certs\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.895911 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.895815    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istiod-ca-cert\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.895911 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.895839    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5v2k\" (UniqueName: \"kubernetes.io/projected/5e60eb51-e478-4cdc-b604-98e106f6bb4f-kube-api-access-n5v2k\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.895911 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.895872    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-credential-socket\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.896145 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.895963    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-workload-socket\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.896145 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.896109    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-data\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.896310 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.896284    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-workload-certs\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.896513 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.896413    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-credential-socket\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.896626 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.896602    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istiod-ca-cert\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.898013 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.897992    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-envoy\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.898451 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.898434    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-podinfo\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.919018 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.918990    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-token\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:03.919305 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:03.919281    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5v2k\" (UniqueName: \"kubernetes.io/projected/5e60eb51-e478-4cdc-b604-98e106f6bb4f-kube-api-access-n5v2k\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:04.018409 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:04.018374    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:04.352476 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:04.352449    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"]
Oct 30 01:05:04.353750 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:05:04.353727    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e60eb51_e478_4cdc_b604_98e106f6bb4f.slice/crio-d310e28d9502b5bdba2cd4b3e99d28ab301196afd026f272783efe08c86718f4 WatchSource:0}: Error finding container d310e28d9502b5bdba2cd4b3e99d28ab301196afd026f272783efe08c86718f4: Status 404 returned error can't find the container with id d310e28d9502b5bdba2cd4b3e99d28ab301196afd026f272783efe08c86718f4
Oct 30 01:05:04.356284 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:04.356252    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:05:04.356350 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:04.356319    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:05:04.356396 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:04.356351    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:05:04.578884 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:04.578846    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"]
Oct 30 01:05:04.579091 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:04.579053    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h" podUID="b2e74574-f055-44f7-b920-ef38942e05c5" containerName="istio-proxy" containerID="cri-o://4c549b18a76030ee6b3c03ad2ae308b1e4f643f7e56ffd92929ca3863e864072" gracePeriod=30
Oct 30 01:05:04.717236 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:04.717197    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq" event={"ID":"5e60eb51-e478-4cdc-b604-98e106f6bb4f","Type":"ContainerStarted","Data":"a11e7709cdf53c68bfa941109c6a1b550dd5a0c779fdd6b379b10ca4b774648c"}
Oct 30 01:05:04.717734 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:04.717264    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq" event={"ID":"5e60eb51-e478-4cdc-b604-98e106f6bb4f","Type":"ContainerStarted","Data":"d310e28d9502b5bdba2cd4b3e99d28ab301196afd026f272783efe08c86718f4"}
Oct 30 01:05:04.775838 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:04.775764    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq" podStartSLOduration=1.7757396540000001 podStartE2EDuration="1.775739654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:04.770560941 +0000 UTC m=+1417.108662870" watchObservedRunningTime="2026-10-30 01:05:04.775739654 +0000 UTC m=+1417.113841564"
Oct 30 01:05:05.019167 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:05.019088    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:05.020532 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:05.020504    4864 patch_prober.go:29] interesting pod/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 01:05:05.020673 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:05.020565    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq" podUID="5e60eb51-e478-4cdc-b604-98e106f6bb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 01:05:06.018966 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.018931    4864 patch_prober.go:29] interesting pod/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 01:05:06.019394 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.019011    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq" podUID="5e60eb51-e478-4cdc-b604-98e106f6bb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 01:05:06.190832 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.190806    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:05:06.334567 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.334530    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-kube-api-access-jm2bq\" (UniqueName: \"kubernetes.io/projected/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-kube-api-access-jm2bq\") pod \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") "
Oct 30 01:05:06.334742 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.334604    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-data\") pod \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") "
Oct 30 01:05:06.334742 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.334664    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-workload-certs\") pod \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") "
Oct 30 01:05:06.334832 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.334816    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-envoy\") pod \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") "
Oct 30 01:05:06.334892 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.334871    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-token\" (UniqueName: \"kubernetes.io/projected/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-token\") pod \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") "
Oct 30 01:05:06.334941 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.334892    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-data" pod "e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" (UID: "e27f9baa-b6fc-4d5b-9c49-56e93c2a703b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:06.334941 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.334910    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-credential-socket\") pod \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") "
Oct 30 01:05:06.335049 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.334959    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-workload-certs" pod "e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" (UID: "e27f9baa-b6fc-4d5b-9c49-56e93c2a703b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:06.335049 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.334979    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-workload-socket\") pod \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") "
Oct 30 01:05:06.335170 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.335128    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istiod-ca-cert\") pod \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") "
Oct 30 01:05:06.335235 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.335168    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-podinfo\") pod \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\" (UID: \"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b\") "
Oct 30 01:05:06.335235 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.335169    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-credential-socket" pod "e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" (UID: "e27f9baa-b6fc-4d5b-9c49-56e93c2a703b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:06.335342 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.335248    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-workload-socket" pod "e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" (UID: "e27f9baa-b6fc-4d5b-9c49-56e93c2a703b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:06.335542 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.335476    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istiod-ca-cert" pod "e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" (UID: "e27f9baa-b6fc-4d5b-9c49-56e93c2a703b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:06.335884 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.335864    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:06.335979 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.335888    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:06.335979 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.335901    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:06.335979 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.335914    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:06.335979 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.335925    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:06.337034 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.337006    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-kube-api-access-jm2bq" pod "e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" (UID: "e27f9baa-b6fc-4d5b-9c49-56e93c2a703b"). InnerVolumeSpecName "kube-api-access-jm2bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:06.337378 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.337357    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-envoy" pod "e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" (UID: "e27f9baa-b6fc-4d5b-9c49-56e93c2a703b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:06.337462 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.337446    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-token" pod "e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" (UID: "e27f9baa-b6fc-4d5b-9c49-56e93c2a703b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:06.337539 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.337521    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-podinfo" pod "e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" (UID: "e27f9baa-b6fc-4d5b-9c49-56e93c2a703b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:05:06.437287 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.437208    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:06.437287 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.437236    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:06.437287 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.437245    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:06.437287 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.437254    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm2bq\" (UniqueName: \"kubernetes.io/projected/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b-kube-api-access-jm2bq\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:06.732761 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.732722    4864 generic.go:356] "Generic (PLEG): container finished" podID="e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" containerID="74d8065112360c40433132ec829ea281a81c0ddad460bc2d28b9b54f7a76242b" exitCode=0
Oct 30 01:05:06.732955 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.732777    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"
Oct 30 01:05:06.732955 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.732801    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt" event={"ID":"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b","Type":"ContainerDied","Data":"74d8065112360c40433132ec829ea281a81c0ddad460bc2d28b9b54f7a76242b"}
Oct 30 01:05:06.732955 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.732854    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt" event={"ID":"e27f9baa-b6fc-4d5b-9c49-56e93c2a703b","Type":"ContainerDied","Data":"9f5be449f4fa0cfe9d96d7878e5a572a874d40765679877b5b32e1d64f8eb46e"}
Oct 30 01:05:06.732955 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.732877    4864 scope.go:122] "RemoveContainer" containerID="74d8065112360c40433132ec829ea281a81c0ddad460bc2d28b9b54f7a76242b"
Oct 30 01:05:06.744388 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.744285    4864 scope.go:122] "RemoveContainer" containerID="c6d4f8300c978be5d76ad8699a85f890b7f3f23b49a3f3cbad9965ba4a19d487"
Oct 30 01:05:06.753981 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.753964    4864 scope.go:122] "RemoveContainer" containerID="74d8065112360c40433132ec829ea281a81c0ddad460bc2d28b9b54f7a76242b"
Oct 30 01:05:06.754294 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:05:06.754270    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74d8065112360c40433132ec829ea281a81c0ddad460bc2d28b9b54f7a76242b\": container with ID starting with 74d8065112360c40433132ec829ea281a81c0ddad460bc2d28b9b54f7a76242b not found: ID does not exist" containerID="74d8065112360c40433132ec829ea281a81c0ddad460bc2d28b9b54f7a76242b"
Oct 30 01:05:06.754410 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.754300    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74d8065112360c40433132ec829ea281a81c0ddad460bc2d28b9b54f7a76242b"} err="failed to get container status \"74d8065112360c40433132ec829ea281a81c0ddad460bc2d28b9b54f7a76242b\": rpc error: code = NotFound desc = could not find container \"74d8065112360c40433132ec829ea281a81c0ddad460bc2d28b9b54f7a76242b\": container with ID starting with 74d8065112360c40433132ec829ea281a81c0ddad460bc2d28b9b54f7a76242b not found: ID does not exist"
Oct 30 01:05:06.754410 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.754325    4864 scope.go:122] "RemoveContainer" containerID="c6d4f8300c978be5d76ad8699a85f890b7f3f23b49a3f3cbad9965ba4a19d487"
Oct 30 01:05:06.754562 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:05:06.754544    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6d4f8300c978be5d76ad8699a85f890b7f3f23b49a3f3cbad9965ba4a19d487\": container with ID starting with c6d4f8300c978be5d76ad8699a85f890b7f3f23b49a3f3cbad9965ba4a19d487 not found: ID does not exist" containerID="c6d4f8300c978be5d76ad8699a85f890b7f3f23b49a3f3cbad9965ba4a19d487"
Oct 30 01:05:06.754606 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.754566    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6d4f8300c978be5d76ad8699a85f890b7f3f23b49a3f3cbad9965ba4a19d487"} err="failed to get container status \"c6d4f8300c978be5d76ad8699a85f890b7f3f23b49a3f3cbad9965ba4a19d487\": rpc error: code = NotFound desc = could not find container \"c6d4f8300c978be5d76ad8699a85f890b7f3f23b49a3f3cbad9965ba4a19d487\": container with ID starting with c6d4f8300c978be5d76ad8699a85f890b7f3f23b49a3f3cbad9965ba4a19d487 not found: ID does not exist"
Oct 30 01:05:06.800680 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.800648    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"]
Oct 30 01:05:06.805329 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:06.805303    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9b2c3f-openshift-default-7b4c748f7f-wstxt"]
Oct 30 01:05:07.019627 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:07.019541    4864 patch_prober.go:29] interesting pod/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 01:05:07.019627 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:07.019606    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq" podUID="5e60eb51-e478-4cdc-b604-98e106f6bb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 01:05:07.799705 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:07.799668    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" path="/var/lib/kubelet/pods/e27f9baa-b6fc-4d5b-9c49-56e93c2a703b/volumes"
Oct 30 01:05:08.018745 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:08.018711    4864 patch_prober.go:29] interesting pod/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 01:05:08.018942 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:08.018765    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq" podUID="5e60eb51-e478-4cdc-b604-98e106f6bb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 01:05:09.018999 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.018968    4864 patch_prober.go:29] interesting pod/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 01:05:09.019415 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.019033    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq" podUID="5e60eb51-e478-4cdc-b604-98e106f6bb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 01:05:09.755845 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.755813    4864 generic.go:356] "Generic (PLEG): container finished" podID="b2e74574-f055-44f7-b920-ef38942e05c5" containerID="4c549b18a76030ee6b3c03ad2ae308b1e4f643f7e56ffd92929ca3863e864072" exitCode=0
Oct 30 01:05:09.756156 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.755886    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h" event={"ID":"b2e74574-f055-44f7-b920-ef38942e05c5","Type":"ContainerDied","Data":"4c549b18a76030ee6b3c03ad2ae308b1e4f643f7e56ffd92929ca3863e864072"}
Oct 30 01:05:09.917295 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.917243    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 01:05:09.932359 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.932335    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e74574-f055-44f7-b920-ef38942e05c5-kube-api-access-nkdp5\" (UniqueName: \"kubernetes.io/projected/b2e74574-f055-44f7-b920-ef38942e05c5-kube-api-access-nkdp5\") pod \"b2e74574-f055-44f7-b920-ef38942e05c5\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") "
Oct 30 01:05:09.932544 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.932390    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-istio-data\") pod \"b2e74574-f055-44f7-b920-ef38942e05c5\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") "
Oct 30 01:05:09.932544 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.932431    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-credential-socket\") pod \"b2e74574-f055-44f7-b920-ef38942e05c5\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") "
Oct 30 01:05:09.932544 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.932465    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-workload-certs\") pod \"b2e74574-f055-44f7-b920-ef38942e05c5\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") "
Oct 30 01:05:09.932544 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.932485    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e74574-f055-44f7-b920-ef38942e05c5-istio-token\" (UniqueName: \"kubernetes.io/projected/b2e74574-f055-44f7-b920-ef38942e05c5-istio-token\") pod \"b2e74574-f055-44f7-b920-ef38942e05c5\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") "
Oct 30 01:05:09.932544 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.932525    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2e74574-f055-44f7-b920-ef38942e05c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e74574-f055-44f7-b920-ef38942e05c5-istiod-ca-cert\") pod \"b2e74574-f055-44f7-b920-ef38942e05c5\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") "
Oct 30 01:05:09.932700 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.932563    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-istio-envoy\") pod \"b2e74574-f055-44f7-b920-ef38942e05c5\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") "
Oct 30 01:05:09.932700 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.932599    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2e74574-f055-44f7-b920-ef38942e05c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e74574-f055-44f7-b920-ef38942e05c5-istio-podinfo\") pod \"b2e74574-f055-44f7-b920-ef38942e05c5\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") "
Oct 30 01:05:09.932700 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.932616    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-workload-socket\") pod \"b2e74574-f055-44f7-b920-ef38942e05c5\" (UID: \"b2e74574-f055-44f7-b920-ef38942e05c5\") "
Oct 30 01:05:09.932826 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.932728    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-istio-data" pod "b2e74574-f055-44f7-b920-ef38942e05c5" (UID: "b2e74574-f055-44f7-b920-ef38942e05c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:09.932826 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.932740    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-credential-socket" pod "b2e74574-f055-44f7-b920-ef38942e05c5" (UID: "b2e74574-f055-44f7-b920-ef38942e05c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:09.932928 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.932889    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e74574-f055-44f7-b920-ef38942e05c5-istiod-ca-cert" pod "b2e74574-f055-44f7-b920-ef38942e05c5" (UID: "b2e74574-f055-44f7-b920-ef38942e05c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:09.933265 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.933098    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-workload-socket" pod "b2e74574-f055-44f7-b920-ef38942e05c5" (UID: "b2e74574-f055-44f7-b920-ef38942e05c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:09.933265 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.933231    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-workload-certs" pod "b2e74574-f055-44f7-b920-ef38942e05c5" (UID: "b2e74574-f055-44f7-b920-ef38942e05c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:09.933443 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.933300    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:09.933443 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.933338    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:09.933443 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.933353    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e74574-f055-44f7-b920-ef38942e05c5-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:09.933443 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.933364    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:09.935065 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.935038    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-istio-envoy" pod "b2e74574-f055-44f7-b920-ef38942e05c5" (UID: "b2e74574-f055-44f7-b920-ef38942e05c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:09.935193 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.935095    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e74574-f055-44f7-b920-ef38942e05c5-istio-token" pod "b2e74574-f055-44f7-b920-ef38942e05c5" (UID: "b2e74574-f055-44f7-b920-ef38942e05c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:09.935292 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.935272    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2e74574-f055-44f7-b920-ef38942e05c5-istio-podinfo" pod "b2e74574-f055-44f7-b920-ef38942e05c5" (UID: "b2e74574-f055-44f7-b920-ef38942e05c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:05:09.935979 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:09.935960    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e74574-f055-44f7-b920-ef38942e05c5-kube-api-access-nkdp5" pod "b2e74574-f055-44f7-b920-ef38942e05c5" (UID: "b2e74574-f055-44f7-b920-ef38942e05c5"). InnerVolumeSpecName "kube-api-access-nkdp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:10.018795 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:10.018765    4864 patch_prober.go:29] interesting pod/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 01:05:10.019028 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:10.019006    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq" podUID="5e60eb51-e478-4cdc-b604-98e106f6bb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 01:05:10.034919 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:10.034887    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.035138 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:10.035123    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e74574-f055-44f7-b920-ef38942e05c5-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.035873 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:10.035850    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkdp5\" (UniqueName: \"kubernetes.io/projected/b2e74574-f055-44f7-b920-ef38942e05c5-kube-api-access-nkdp5\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.035987 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:10.035977    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e74574-f055-44f7-b920-ef38942e05c5-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.036052 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:10.036045    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e74574-f055-44f7-b920-ef38942e05c5-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.803095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:10.791956    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h" event={"ID":"b2e74574-f055-44f7-b920-ef38942e05c5","Type":"ContainerDied","Data":"36159c11fc5dc8e89bf26cae0a626dbc4a8861906e955408b364065fbd79f6ea"}
Oct 30 01:05:10.803095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:10.792012    4864 scope.go:122] "RemoveContainer" containerID="4c549b18a76030ee6b3c03ad2ae308b1e4f643f7e56ffd92929ca3863e864072"
Oct 30 01:05:10.803095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:10.792158    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"
Oct 30 01:05:10.864098 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:10.857041    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"]
Oct 30 01:05:10.889093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:10.888740    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-vgb7h"]
Oct 30 01:05:11.019427 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:11.019388    4864 patch_prober.go:29] interesting pod/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 01:05:11.019839 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:11.019457    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq" podUID="5e60eb51-e478-4cdc-b604-98e106f6bb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 01:05:11.795966 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:11.795933    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2e74574-f055-44f7-b920-ef38942e05c5" path="/var/lib/kubelet/pods/b2e74574-f055-44f7-b920-ef38942e05c5/volumes"
Oct 30 01:05:12.023223 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:12.023195    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:12.023641 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:12.023621    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:12.024455 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:12.024437    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:05:15.900940 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:15.900901    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"]
Oct 30 01:05:15.901447 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:15.901215    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss" podUID="04553b1e-6cf7-498d-8fab-b282992c70f3" containerName="istio-proxy" containerID="cri-o://a15c95a0cb4a1cf71345b62150c4268c1b70aaf61c7f6796d721f8b9e4474061" gracePeriod=30
Oct 30 01:05:16.042529 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:16.042483    4864 patch_prober.go:29] interesting pod/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:05:16.042884 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:16.042843    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss" podUID="04553b1e-6cf7-498d-8fab-b282992c70f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:05:18.611652 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:18.611617    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv"]
Oct 30 01:05:19.041459 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:05:19.041422    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:05:21.193369 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.193340    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:05:21.271278 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.271240    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-workload-certs\") pod \"04553b1e-6cf7-498d-8fab-b282992c70f3\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") "
Oct 30 01:05:21.271482 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.271293    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-podinfo\") pod \"04553b1e-6cf7-498d-8fab-b282992c70f3\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") "
Oct 30 01:05:21.271482 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.271317    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04553b1e-6cf7-498d-8fab-b282992c70f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04553b1e-6cf7-498d-8fab-b282992c70f3-istiod-ca-cert\") pod \"04553b1e-6cf7-498d-8fab-b282992c70f3\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") "
Oct 30 01:05:21.271482 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.271353    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04553b1e-6cf7-498d-8fab-b282992c70f3-kube-api-access-r256q\" (UniqueName: \"kubernetes.io/projected/04553b1e-6cf7-498d-8fab-b282992c70f3-kube-api-access-r256q\") pod \"04553b1e-6cf7-498d-8fab-b282992c70f3\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") "
Oct 30 01:05:21.271482 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.271379    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-token\" (UniqueName: \"kubernetes.io/projected/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-token\") pod \"04553b1e-6cf7-498d-8fab-b282992c70f3\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") "
Oct 30 01:05:21.271714 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.271614    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-workload-certs" pod "04553b1e-6cf7-498d-8fab-b282992c70f3" (UID: "04553b1e-6cf7-498d-8fab-b282992c70f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:21.271812 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.271786    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04553b1e-6cf7-498d-8fab-b282992c70f3-istiod-ca-cert" pod "04553b1e-6cf7-498d-8fab-b282992c70f3" (UID: "04553b1e-6cf7-498d-8fab-b282992c70f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:21.272910 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.272035    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-credential-socket\") pod \"04553b1e-6cf7-498d-8fab-b282992c70f3\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") "
Oct 30 01:05:21.272910 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.272149    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-envoy\") pod \"04553b1e-6cf7-498d-8fab-b282992c70f3\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") "
Oct 30 01:05:21.272910 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.272211    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-data\") pod \"04553b1e-6cf7-498d-8fab-b282992c70f3\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") "
Oct 30 01:05:21.272910 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.272298    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-credential-socket" pod "04553b1e-6cf7-498d-8fab-b282992c70f3" (UID: "04553b1e-6cf7-498d-8fab-b282992c70f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:21.272910 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.272431    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-workload-socket\") pod \"04553b1e-6cf7-498d-8fab-b282992c70f3\" (UID: \"04553b1e-6cf7-498d-8fab-b282992c70f3\") "
Oct 30 01:05:21.272910 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.272681    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-data" pod "04553b1e-6cf7-498d-8fab-b282992c70f3" (UID: "04553b1e-6cf7-498d-8fab-b282992c70f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:21.273297 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.273159    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:21.273297 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.273182    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04553b1e-6cf7-498d-8fab-b282992c70f3-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:21.273297 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.273195    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:21.273297 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.273210    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:21.273568 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.273544    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-workload-socket" pod "04553b1e-6cf7-498d-8fab-b282992c70f3" (UID: "04553b1e-6cf7-498d-8fab-b282992c70f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:21.274226 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.274194    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-podinfo" pod "04553b1e-6cf7-498d-8fab-b282992c70f3" (UID: "04553b1e-6cf7-498d-8fab-b282992c70f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:05:21.274320 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.274298    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04553b1e-6cf7-498d-8fab-b282992c70f3-kube-api-access-r256q" pod "04553b1e-6cf7-498d-8fab-b282992c70f3" (UID: "04553b1e-6cf7-498d-8fab-b282992c70f3"). InnerVolumeSpecName "kube-api-access-r256q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:21.274375 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.274362    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-token" pod "04553b1e-6cf7-498d-8fab-b282992c70f3" (UID: "04553b1e-6cf7-498d-8fab-b282992c70f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:21.275233 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.275215    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-envoy" pod "04553b1e-6cf7-498d-8fab-b282992c70f3" (UID: "04553b1e-6cf7-498d-8fab-b282992c70f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:21.374500 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.374465    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:21.374500 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.374491    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:21.374500 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.374501    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r256q\" (UniqueName: \"kubernetes.io/projected/04553b1e-6cf7-498d-8fab-b282992c70f3-kube-api-access-r256q\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:21.374728 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.374514    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:21.374728 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.374526    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04553b1e-6cf7-498d-8fab-b282992c70f3-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:21.909217 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.909177    4864 generic.go:356] "Generic (PLEG): container finished" podID="04553b1e-6cf7-498d-8fab-b282992c70f3" containerID="a15c95a0cb4a1cf71345b62150c4268c1b70aaf61c7f6796d721f8b9e4474061" exitCode=0
Oct 30 01:05:21.909406 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.909256    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss" event={"ID":"04553b1e-6cf7-498d-8fab-b282992c70f3","Type":"ContainerDied","Data":"a15c95a0cb4a1cf71345b62150c4268c1b70aaf61c7f6796d721f8b9e4474061"}
Oct 30 01:05:21.909406 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.909273    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"
Oct 30 01:05:21.909406 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.909317    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss" event={"ID":"04553b1e-6cf7-498d-8fab-b282992c70f3","Type":"ContainerDied","Data":"7d19a720a9d6cb39f6d1b943ffeea8bfa941d14a3ea8da733ea0eee1801b7f4d"}
Oct 30 01:05:21.909406 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.909341    4864 scope.go:122] "RemoveContainer" containerID="a15c95a0cb4a1cf71345b62150c4268c1b70aaf61c7f6796d721f8b9e4474061"
Oct 30 01:05:21.939330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.939298    4864 scope.go:122] "RemoveContainer" containerID="a15c95a0cb4a1cf71345b62150c4268c1b70aaf61c7f6796d721f8b9e4474061"
Oct 30 01:05:21.942537 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:05:21.942506    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a15c95a0cb4a1cf71345b62150c4268c1b70aaf61c7f6796d721f8b9e4474061\": container with ID starting with a15c95a0cb4a1cf71345b62150c4268c1b70aaf61c7f6796d721f8b9e4474061 not found: ID does not exist" containerID="a15c95a0cb4a1cf71345b62150c4268c1b70aaf61c7f6796d721f8b9e4474061"
Oct 30 01:05:21.942688 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.942546    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a15c95a0cb4a1cf71345b62150c4268c1b70aaf61c7f6796d721f8b9e4474061"} err="failed to get container status \"a15c95a0cb4a1cf71345b62150c4268c1b70aaf61c7f6796d721f8b9e4474061\": rpc error: code = NotFound desc = could not find container \"a15c95a0cb4a1cf71345b62150c4268c1b70aaf61c7f6796d721f8b9e4474061\": container with ID starting with a15c95a0cb4a1cf71345b62150c4268c1b70aaf61c7f6796d721f8b9e4474061 not found: ID does not exist"
Oct 30 01:05:21.974728 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.974693    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"]
Oct 30 01:05:21.983888 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:21.983855    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8363a40b-openshift-default-677fff8ff7-j29ss"]
Oct 30 01:05:23.612517 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:23.612470    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv" podUID="bea59237-06d5-4067-8057-1b87d5dd7f4c" containerName="payload-processing" containerID="cri-o://4b8e51b4652f49870959a62a1d9be29fb349b40d7aab3115df17ab629f458665" gracePeriod=25
Oct 30 01:05:23.821205 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:23.821164    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04553b1e-6cf7-498d-8fab-b282992c70f3" path="/var/lib/kubelet/pods/04553b1e-6cf7-498d-8fab-b282992c70f3/volumes"
Oct 30 01:05:23.893794 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:23.893761    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv"
Oct 30 01:05:23.911186 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:23.911154    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bea59237-06d5-4067-8057-1b87d5dd7f4c-kube-api-access-wqnvf\" (UniqueName: \"kubernetes.io/projected/bea59237-06d5-4067-8057-1b87d5dd7f4c-kube-api-access-wqnvf\") pod \"bea59237-06d5-4067-8057-1b87d5dd7f4c\" (UID: \"bea59237-06d5-4067-8057-1b87d5dd7f4c\") "
Oct 30 01:05:23.911336 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:23.911216    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bea59237-06d5-4067-8057-1b87d5dd7f4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea59237-06d5-4067-8057-1b87d5dd7f4c-plugins-config-volume\") pod \"bea59237-06d5-4067-8057-1b87d5dd7f4c\" (UID: \"bea59237-06d5-4067-8057-1b87d5dd7f4c\") "
Oct 30 01:05:23.911605 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:23.911577    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bea59237-06d5-4067-8057-1b87d5dd7f4c-plugins-config-volume" pod "bea59237-06d5-4067-8057-1b87d5dd7f4c" (UID: "bea59237-06d5-4067-8057-1b87d5dd7f4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:23.911855 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:23.911838    4864 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea59237-06d5-4067-8057-1b87d5dd7f4c-plugins-config-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:23.913305 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:23.913286    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bea59237-06d5-4067-8057-1b87d5dd7f4c-kube-api-access-wqnvf" pod "bea59237-06d5-4067-8057-1b87d5dd7f4c" (UID: "bea59237-06d5-4067-8057-1b87d5dd7f4c"). InnerVolumeSpecName "kube-api-access-wqnvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:23.923866 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:23.923843    4864 generic.go:356] "Generic (PLEG): container finished" podID="bea59237-06d5-4067-8057-1b87d5dd7f4c" containerID="4b8e51b4652f49870959a62a1d9be29fb349b40d7aab3115df17ab629f458665" exitCode=0
Oct 30 01:05:23.923962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:23.923898    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv"
Oct 30 01:05:23.923962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:23.923935    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv" event={"ID":"bea59237-06d5-4067-8057-1b87d5dd7f4c","Type":"ContainerDied","Data":"4b8e51b4652f49870959a62a1d9be29fb349b40d7aab3115df17ab629f458665"}
Oct 30 01:05:23.924064 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:23.923997    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv" event={"ID":"bea59237-06d5-4067-8057-1b87d5dd7f4c","Type":"ContainerDied","Data":"ac96e3a083a0c200c09ebd7a7cf09905e5c2aae685d9b5d139141e6ab747271b"}
Oct 30 01:05:23.924064 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:23.924008    4864 scope.go:122] "RemoveContainer" containerID="4b8e51b4652f49870959a62a1d9be29fb349b40d7aab3115df17ab629f458665"
Oct 30 01:05:23.938379 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:23.938356    4864 scope.go:122] "RemoveContainer" containerID="4b8e51b4652f49870959a62a1d9be29fb349b40d7aab3115df17ab629f458665"
Oct 30 01:05:23.938700 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:05:23.938681    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b8e51b4652f49870959a62a1d9be29fb349b40d7aab3115df17ab629f458665\": container with ID starting with 4b8e51b4652f49870959a62a1d9be29fb349b40d7aab3115df17ab629f458665 not found: ID does not exist" containerID="4b8e51b4652f49870959a62a1d9be29fb349b40d7aab3115df17ab629f458665"
Oct 30 01:05:23.938749 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:23.938709    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b8e51b4652f49870959a62a1d9be29fb349b40d7aab3115df17ab629f458665"} err="failed to get container status \"4b8e51b4652f49870959a62a1d9be29fb349b40d7aab3115df17ab629f458665\": rpc error: code = NotFound desc = could not find container \"4b8e51b4652f49870959a62a1d9be29fb349b40d7aab3115df17ab629f458665\": container with ID starting with 4b8e51b4652f49870959a62a1d9be29fb349b40d7aab3115df17ab629f458665 not found: ID does not exist"
Oct 30 01:05:23.979813 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:23.979774    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv"]
Oct 30 01:05:23.985125 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:23.985059    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d4622a-6954f98c4-xbmrv"]
Oct 30 01:05:24.012870 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:24.012832    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqnvf\" (UniqueName: \"kubernetes.io/projected/bea59237-06d5-4067-8057-1b87d5dd7f4c-kube-api-access-wqnvf\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:25.795861 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:25.795827    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bea59237-06d5-4067-8057-1b87d5dd7f4c" path="/var/lib/kubelet/pods/bea59237-06d5-4067-8057-1b87d5dd7f4c/volumes"
Oct 30 01:05:28.962582 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:28.962542    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"]
Oct 30 01:05:28.963146 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:28.963089    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62" podUID="76501308-cde8-41f2-aa4e-b6039b294be5" containerName="istio-proxy" containerID="cri-o://d8eb9024b6b25e44b25d15bd00c7cf401ba0ec874906183dca699262cda1d320" gracePeriod=30
Oct 30 01:05:30.900807 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:30.900771    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5"]
Oct 30 01:05:30.901363 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:30.901023    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5" podUID="fe1b890e-9648-4210-aa76-ef90b4ad983c" containerName="maas-api" containerID="cri-o://4a4a50918d8b3403ef70611582ff40fa0f860078a49f479c820c180637547b04" gracePeriod=30
Oct 30 01:05:31.238225 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:31.238187    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5"
Oct 30 01:05:31.332582 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:31.332543    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe1b890e-9648-4210-aa76-ef90b4ad983c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe1b890e-9648-4210-aa76-ef90b4ad983c-maas-api-tls\") pod \"fe1b890e-9648-4210-aa76-ef90b4ad983c\" (UID: \"fe1b890e-9648-4210-aa76-ef90b4ad983c\") "
Oct 30 01:05:31.332793 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:31.332614    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe1b890e-9648-4210-aa76-ef90b4ad983c-kube-api-access-qmxlc\" (UniqueName: \"kubernetes.io/projected/fe1b890e-9648-4210-aa76-ef90b4ad983c-kube-api-access-qmxlc\") pod \"fe1b890e-9648-4210-aa76-ef90b4ad983c\" (UID: \"fe1b890e-9648-4210-aa76-ef90b4ad983c\") "
Oct 30 01:05:31.332793 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:31.332734    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe1b890e-9648-4210-aa76-ef90b4ad983c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1b890e-9648-4210-aa76-ef90b4ad983c-openshift-service-ca\") pod \"fe1b890e-9648-4210-aa76-ef90b4ad983c\" (UID: \"fe1b890e-9648-4210-aa76-ef90b4ad983c\") "
Oct 30 01:05:31.332993 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:31.332972    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe1b890e-9648-4210-aa76-ef90b4ad983c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe1b890e-9648-4210-aa76-ef90b4ad983c-metrics-certs\") pod \"fe1b890e-9648-4210-aa76-ef90b4ad983c\" (UID: \"fe1b890e-9648-4210-aa76-ef90b4ad983c\") "
Oct 30 01:05:31.335021 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:31.334983    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1b890e-9648-4210-aa76-ef90b4ad983c-openshift-service-ca" pod "fe1b890e-9648-4210-aa76-ef90b4ad983c" (UID: "fe1b890e-9648-4210-aa76-ef90b4ad983c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:31.336983 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:31.336952    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe1b890e-9648-4210-aa76-ef90b4ad983c-kube-api-access-qmxlc" pod "fe1b890e-9648-4210-aa76-ef90b4ad983c" (UID: "fe1b890e-9648-4210-aa76-ef90b4ad983c"). InnerVolumeSpecName "kube-api-access-qmxlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:31.337355 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:31.337330    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1b890e-9648-4210-aa76-ef90b4ad983c-metrics-certs" pod "fe1b890e-9648-4210-aa76-ef90b4ad983c" (UID: "fe1b890e-9648-4210-aa76-ef90b4ad983c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:05:31.337502 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:31.337484    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1b890e-9648-4210-aa76-ef90b4ad983c-maas-api-tls" pod "fe1b890e-9648-4210-aa76-ef90b4ad983c" (UID: "fe1b890e-9648-4210-aa76-ef90b4ad983c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:05:31.436183 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:31.436143    4864 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1b890e-9648-4210-aa76-ef90b4ad983c-openshift-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:31.436183 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:31.436184    4864 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe1b890e-9648-4210-aa76-ef90b4ad983c-metrics-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:31.436449 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:31.436199    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe1b890e-9648-4210-aa76-ef90b4ad983c-maas-api-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:31.436449 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:31.436212    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmxlc\" (UniqueName: \"kubernetes.io/projected/fe1b890e-9648-4210-aa76-ef90b4ad983c-kube-api-access-qmxlc\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:31.981288 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:31.981254    4864 generic.go:356] "Generic (PLEG): container finished" podID="fe1b890e-9648-4210-aa76-ef90b4ad983c" containerID="4a4a50918d8b3403ef70611582ff40fa0f860078a49f479c820c180637547b04" exitCode=0
Oct 30 01:05:31.981698 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:31.981301    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5"
Oct 30 01:05:31.981698 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:31.981338    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5" event={"ID":"fe1b890e-9648-4210-aa76-ef90b4ad983c","Type":"ContainerDied","Data":"4a4a50918d8b3403ef70611582ff40fa0f860078a49f479c820c180637547b04"}
Oct 30 01:05:31.981698 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:31.981394    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5" event={"ID":"fe1b890e-9648-4210-aa76-ef90b4ad983c","Type":"ContainerDied","Data":"d2eb9524c3a1c39267865958671d93f9a88885e06555854e49598e12697c0812"}
Oct 30 01:05:31.981698 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:31.981417    4864 scope.go:122] "RemoveContainer" containerID="4a4a50918d8b3403ef70611582ff40fa0f860078a49f479c820c180637547b04"
Oct 30 01:05:31.991158 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:31.991133    4864 scope.go:122] "RemoveContainer" containerID="4a4a50918d8b3403ef70611582ff40fa0f860078a49f479c820c180637547b04"
Oct 30 01:05:31.991435 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:05:31.991414    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a4a50918d8b3403ef70611582ff40fa0f860078a49f479c820c180637547b04\": container with ID starting with 4a4a50918d8b3403ef70611582ff40fa0f860078a49f479c820c180637547b04 not found: ID does not exist" containerID="4a4a50918d8b3403ef70611582ff40fa0f860078a49f479c820c180637547b04"
Oct 30 01:05:31.991517 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:31.991444    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4a50918d8b3403ef70611582ff40fa0f860078a49f479c820c180637547b04"} err="failed to get container status \"4a4a50918d8b3403ef70611582ff40fa0f860078a49f479c820c180637547b04\": rpc error: code = NotFound desc = could not find container \"4a4a50918d8b3403ef70611582ff40fa0f860078a49f479c820c180637547b04\": container with ID starting with 4a4a50918d8b3403ef70611582ff40fa0f860078a49f479c820c180637547b04 not found: ID does not exist"
Oct 30 01:05:32.029440 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:32.029408    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5"]
Oct 30 01:05:32.039084 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:32.039035    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-645126-58cb667c69-9bpf5"]
Oct 30 01:05:33.793996 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:33.793959    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe1b890e-9648-4210-aa76-ef90b4ad983c" path="/var/lib/kubelet/pods/fe1b890e-9648-4210-aa76-ef90b4ad983c/volumes"
Oct 30 01:05:34.248168 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.248143    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 01:05:34.375493 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.375410    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-istio-data\") pod \"76501308-cde8-41f2-aa4e-b6039b294be5\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") "
Oct 30 01:05:34.375493 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.375444    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-workload-socket\") pod \"76501308-cde8-41f2-aa4e-b6039b294be5\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") "
Oct 30 01:05:34.375722 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.375515    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76501308-cde8-41f2-aa4e-b6039b294be5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76501308-cde8-41f2-aa4e-b6039b294be5-istio-podinfo\") pod \"76501308-cde8-41f2-aa4e-b6039b294be5\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") "
Oct 30 01:05:34.375722 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.375536    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-workload-certs\") pod \"76501308-cde8-41f2-aa4e-b6039b294be5\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") "
Oct 30 01:05:34.375722 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.375656    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76501308-cde8-41f2-aa4e-b6039b294be5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76501308-cde8-41f2-aa4e-b6039b294be5-istiod-ca-cert\") pod \"76501308-cde8-41f2-aa4e-b6039b294be5\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") "
Oct 30 01:05:34.375722 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.375708    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-credential-socket\") pod \"76501308-cde8-41f2-aa4e-b6039b294be5\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") "
Oct 30 01:05:34.375928 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.375756    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-istio-data" pod "76501308-cde8-41f2-aa4e-b6039b294be5" (UID: "76501308-cde8-41f2-aa4e-b6039b294be5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:34.375928 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.375802    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-workload-socket" pod "76501308-cde8-41f2-aa4e-b6039b294be5" (UID: "76501308-cde8-41f2-aa4e-b6039b294be5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:34.375928 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.375842    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76501308-cde8-41f2-aa4e-b6039b294be5-kube-api-access-lmsd7\" (UniqueName: \"kubernetes.io/projected/76501308-cde8-41f2-aa4e-b6039b294be5-kube-api-access-lmsd7\") pod \"76501308-cde8-41f2-aa4e-b6039b294be5\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") "
Oct 30 01:05:34.375928 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.375868    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76501308-cde8-41f2-aa4e-b6039b294be5-istio-token\" (UniqueName: \"kubernetes.io/projected/76501308-cde8-41f2-aa4e-b6039b294be5-istio-token\") pod \"76501308-cde8-41f2-aa4e-b6039b294be5\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") "
Oct 30 01:05:34.375928 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.375893    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-istio-envoy\") pod \"76501308-cde8-41f2-aa4e-b6039b294be5\" (UID: \"76501308-cde8-41f2-aa4e-b6039b294be5\") "
Oct 30 01:05:34.376226 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.376162    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-credential-socket" pod "76501308-cde8-41f2-aa4e-b6039b294be5" (UID: "76501308-cde8-41f2-aa4e-b6039b294be5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:34.377213 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.376390    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-workload-certs" pod "76501308-cde8-41f2-aa4e-b6039b294be5" (UID: "76501308-cde8-41f2-aa4e-b6039b294be5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:34.377213 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.376597    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.377213 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.376618    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.377213 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.376631    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.377213 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.376645    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.377213 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.376988    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76501308-cde8-41f2-aa4e-b6039b294be5-istiod-ca-cert" pod "76501308-cde8-41f2-aa4e-b6039b294be5" (UID: "76501308-cde8-41f2-aa4e-b6039b294be5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:34.379053 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.379020    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76501308-cde8-41f2-aa4e-b6039b294be5-kube-api-access-lmsd7" pod "76501308-cde8-41f2-aa4e-b6039b294be5" (UID: "76501308-cde8-41f2-aa4e-b6039b294be5"). InnerVolumeSpecName "kube-api-access-lmsd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:34.379053 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.379026    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76501308-cde8-41f2-aa4e-b6039b294be5-istio-podinfo" pod "76501308-cde8-41f2-aa4e-b6039b294be5" (UID: "76501308-cde8-41f2-aa4e-b6039b294be5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:05:34.379382 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.379363    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-istio-envoy" pod "76501308-cde8-41f2-aa4e-b6039b294be5" (UID: "76501308-cde8-41f2-aa4e-b6039b294be5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:34.380116 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.380096    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76501308-cde8-41f2-aa4e-b6039b294be5-istio-token" pod "76501308-cde8-41f2-aa4e-b6039b294be5" (UID: "76501308-cde8-41f2-aa4e-b6039b294be5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:34.481805 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.481735    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76501308-cde8-41f2-aa4e-b6039b294be5-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.481805 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.481804    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmsd7\" (UniqueName: \"kubernetes.io/projected/76501308-cde8-41f2-aa4e-b6039b294be5-kube-api-access-lmsd7\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.481805 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.481816    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76501308-cde8-41f2-aa4e-b6039b294be5-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.482156 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.481827    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76501308-cde8-41f2-aa4e-b6039b294be5-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.482156 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:34.481836    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76501308-cde8-41f2-aa4e-b6039b294be5-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:35.005417 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:35.005377    4864 generic.go:356] "Generic (PLEG): container finished" podID="76501308-cde8-41f2-aa4e-b6039b294be5" containerID="d8eb9024b6b25e44b25d15bd00c7cf401ba0ec874906183dca699262cda1d320" exitCode=0
Oct 30 01:05:35.005858 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:35.005420    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"
Oct 30 01:05:35.005858 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:35.005451    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62" event={"ID":"76501308-cde8-41f2-aa4e-b6039b294be5","Type":"ContainerDied","Data":"d8eb9024b6b25e44b25d15bd00c7cf401ba0ec874906183dca699262cda1d320"}
Oct 30 01:05:35.005858 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:35.005508    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62" event={"ID":"76501308-cde8-41f2-aa4e-b6039b294be5","Type":"ContainerDied","Data":"5a49ae7540aad559af2607523ea7edcdae0220db820e7294392aa4de33fcdc26"}
Oct 30 01:05:35.005858 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:35.005533    4864 scope.go:122] "RemoveContainer" containerID="d8eb9024b6b25e44b25d15bd00c7cf401ba0ec874906183dca699262cda1d320"
Oct 30 01:05:35.017560 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:35.017535    4864 scope.go:122] "RemoveContainer" containerID="d8eb9024b6b25e44b25d15bd00c7cf401ba0ec874906183dca699262cda1d320"
Oct 30 01:05:35.017884 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:05:35.017860    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8eb9024b6b25e44b25d15bd00c7cf401ba0ec874906183dca699262cda1d320\": container with ID starting with d8eb9024b6b25e44b25d15bd00c7cf401ba0ec874906183dca699262cda1d320 not found: ID does not exist" containerID="d8eb9024b6b25e44b25d15bd00c7cf401ba0ec874906183dca699262cda1d320"
Oct 30 01:05:35.017967 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:35.017893    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8eb9024b6b25e44b25d15bd00c7cf401ba0ec874906183dca699262cda1d320"} err="failed to get container status \"d8eb9024b6b25e44b25d15bd00c7cf401ba0ec874906183dca699262cda1d320\": rpc error: code = NotFound desc = could not find container \"d8eb9024b6b25e44b25d15bd00c7cf401ba0ec874906183dca699262cda1d320\": container with ID starting with d8eb9024b6b25e44b25d15bd00c7cf401ba0ec874906183dca699262cda1d320 not found: ID does not exist"
Oct 30 01:05:35.063748 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:35.063717    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"]
Oct 30 01:05:35.073411 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:35.073378    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-9tf62"]
Oct 30 01:05:35.794649 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:35.794601    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76501308-cde8-41f2-aa4e-b6039b294be5" path="/var/lib/kubelet/pods/76501308-cde8-41f2-aa4e-b6039b294be5/volumes"
Oct 30 01:05:36.692106 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:36.692048    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"]
Oct 30 01:05:36.692548 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:36.692340    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z" podUID="2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f" containerName="istio-proxy" containerID="cri-o://fb85388065591f9af59ba4981e0a68c69819c554ec03d5df41ce90ffb0854875" gracePeriod=30
Oct 30 01:05:41.963241 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:41.963177    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 01:05:42.009206 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.009165    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-kube-api-access-2xtqn\" (UniqueName: \"kubernetes.io/projected/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-kube-api-access-2xtqn\") pod \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") "
Oct 30 01:05:42.009400 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.009272    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-token\" (UniqueName: \"kubernetes.io/projected/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-token\") pod \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") "
Oct 30 01:05:42.009400 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.009329    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-workload-certs\") pod \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") "
Oct 30 01:05:42.009400 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.009363    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istiod-ca-cert\") pod \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") "
Oct 30 01:05:42.009560 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.009418    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-data\") pod \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") "
Oct 30 01:05:42.009560 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.009510    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-podinfo\") pod \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") "
Oct 30 01:05:42.009560 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.009545    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-workload-socket\") pod \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") "
Oct 30 01:05:42.009712 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.009594    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-credential-socket\") pod \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") "
Oct 30 01:05:42.009712 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.009647    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-workload-certs" pod "2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f" (UID: "2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:42.009712 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.009667    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-envoy\") pod \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\" (UID: \"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f\") "
Oct 30 01:05:42.009869 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.009764    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-data" pod "2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f" (UID: "2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:42.009950 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.009916    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-workload-socket" pod "2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f" (UID: "2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:42.010022 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.010002    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-credential-socket" pod "2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f" (UID: "2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:42.010022 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.009937    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istiod-ca-cert" pod "2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f" (UID: "2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:42.010547 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.010509    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:42.010667 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.010551    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:42.010667 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.010568    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:42.010667 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.010582    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:42.010667 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.010596    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:42.011732 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.011704    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-podinfo" pod "2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f" (UID: "2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:05:42.011833 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.011764    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-kube-api-access-2xtqn" pod "2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f" (UID: "2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f"). InnerVolumeSpecName "kube-api-access-2xtqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:42.011889 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.011868    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-token" pod "2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f" (UID: "2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:42.012104 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.012067    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-envoy" pod "2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f" (UID: "2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:42.061004 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.060953    4864 generic.go:356] "Generic (PLEG): container finished" podID="2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f" containerID="fb85388065591f9af59ba4981e0a68c69819c554ec03d5df41ce90ffb0854875" exitCode=0
Oct 30 01:05:42.061210 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.061101    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"
Oct 30 01:05:42.061210 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.061121    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z" event={"ID":"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f","Type":"ContainerDied","Data":"fb85388065591f9af59ba4981e0a68c69819c554ec03d5df41ce90ffb0854875"}
Oct 30 01:05:42.061210 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.061201    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z" event={"ID":"2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f","Type":"ContainerDied","Data":"db55b13497aa3ac876282c31facc676bbcd118a8f21b4f43f6bf6797bcfe30c2"}
Oct 30 01:05:42.061336 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.061229    4864 scope.go:122] "RemoveContainer" containerID="fb85388065591f9af59ba4981e0a68c69819c554ec03d5df41ce90ffb0854875"
Oct 30 01:05:42.073320 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.073289    4864 scope.go:122] "RemoveContainer" containerID="fb85388065591f9af59ba4981e0a68c69819c554ec03d5df41ce90ffb0854875"
Oct 30 01:05:42.073684 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:05:42.073660    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb85388065591f9af59ba4981e0a68c69819c554ec03d5df41ce90ffb0854875\": container with ID starting with fb85388065591f9af59ba4981e0a68c69819c554ec03d5df41ce90ffb0854875 not found: ID does not exist" containerID="fb85388065591f9af59ba4981e0a68c69819c554ec03d5df41ce90ffb0854875"
Oct 30 01:05:42.073838 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.073692    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb85388065591f9af59ba4981e0a68c69819c554ec03d5df41ce90ffb0854875"} err="failed to get container status \"fb85388065591f9af59ba4981e0a68c69819c554ec03d5df41ce90ffb0854875\": rpc error: code = NotFound desc = could not find container \"fb85388065591f9af59ba4981e0a68c69819c554ec03d5df41ce90ffb0854875\": container with ID starting with fb85388065591f9af59ba4981e0a68c69819c554ec03d5df41ce90ffb0854875 not found: ID does not exist"
Oct 30 01:05:42.111721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.111688    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xtqn\" (UniqueName: \"kubernetes.io/projected/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-kube-api-access-2xtqn\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:42.111721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.111717    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:42.111721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.111730    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:42.111959 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.111739    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:42.123332 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.123299    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"]
Oct 30 01:05:42.130666 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:42.130628    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-l424z"]
Oct 30 01:05:43.794567 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:43.794525    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f" path="/var/lib/kubelet/pods/2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f/volumes"
Oct 30 01:05:48.829413 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.829376    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"]
Oct 30 01:05:48.830141 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.830091    4864 cpu_manager.go:414] "containerMap: removing container" podUID="bea59237-06d5-4067-8057-1b87d5dd7f4c" containerName="payload-processing"
Oct 30 01:05:48.830141 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.830113    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bea59237-06d5-4067-8057-1b87d5dd7f4c" containerName="payload-processing"
Oct 30 01:05:48.830141 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.830132    4864 cpu_manager.go:414] "containerMap: removing container" podUID="fe1b890e-9648-4210-aa76-ef90b4ad983c" containerName="maas-api"
Oct 30 01:05:48.830141 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.830140    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe1b890e-9648-4210-aa76-ef90b4ad983c" containerName="maas-api"
Oct 30 01:05:48.830401 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.830151    4864 cpu_manager.go:414] "containerMap: removing container" podUID="76501308-cde8-41f2-aa4e-b6039b294be5" containerName="istio-proxy"
Oct 30 01:05:48.830401 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.830159    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76501308-cde8-41f2-aa4e-b6039b294be5" containerName="istio-proxy"
Oct 30 01:05:48.830401 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.830186    4864 cpu_manager.go:414] "containerMap: removing container" podUID="04553b1e-6cf7-498d-8fab-b282992c70f3" containerName="istio-proxy"
Oct 30 01:05:48.830401 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.830194    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04553b1e-6cf7-498d-8fab-b282992c70f3" containerName="istio-proxy"
Oct 30 01:05:48.830401 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.830219    4864 cpu_manager.go:414] "containerMap: removing container" podUID="2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f" containerName="istio-proxy"
Oct 30 01:05:48.830401 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.830226    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f" containerName="istio-proxy"
Oct 30 01:05:48.830401 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.830244    4864 cpu_manager.go:414] "containerMap: removing container" podUID="e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" containerName="istio-proxy"
Oct 30 01:05:48.830401 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.830251    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" containerName="istio-proxy"
Oct 30 01:05:48.830401 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.830271    4864 cpu_manager.go:414] "containerMap: removing container" podUID="b2e74574-f055-44f7-b920-ef38942e05c5" containerName="istio-proxy"
Oct 30 01:05:48.830401 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.830279    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e74574-f055-44f7-b920-ef38942e05c5" containerName="istio-proxy"
Oct 30 01:05:48.830882 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.830468    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="04553b1e-6cf7-498d-8fab-b282992c70f3" containerName="istio-proxy"
Oct 30 01:05:48.830882 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.830483    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b4ae8be-3cac-4c96-94a8-a37a9d8cbc5f" containerName="istio-proxy"
Oct 30 01:05:48.830882 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.830492    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" containerName="istio-proxy"
Oct 30 01:05:48.830882 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.830504    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e74574-f055-44f7-b920-ef38942e05c5" containerName="istio-proxy"
Oct 30 01:05:48.830882 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.830513    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" containerName="istio-proxy"
Oct 30 01:05:48.830882 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.830526    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="bea59237-06d5-4067-8057-1b87d5dd7f4c" containerName="payload-processing"
Oct 30 01:05:48.830882 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.830536    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe1b890e-9648-4210-aa76-ef90b4ad983c" containerName="maas-api"
Oct 30 01:05:48.830882 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.830544    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="76501308-cde8-41f2-aa4e-b6039b294be5" containerName="istio-proxy"
Oct 30 01:05:48.833984 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.833905    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:48.861726 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.861695    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"]
Oct 30 01:05:48.912150 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.912115    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b1ff7d-a70f-45d8-8313-1f30c675f356-istiod-ca-cert\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:48.912323 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.912235    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nckmh\" (UniqueName: \"kubernetes.io/projected/71b1ff7d-a70f-45d8-8313-1f30c675f356-kube-api-access-nckmh\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:48.912323 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.912317    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-workload-socket\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:48.912442 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.912414    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-credential-socket\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:48.912500 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.912452    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-podinfo\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:48.912500 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.912480    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-envoy\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:48.912579 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.912563    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-workload-certs\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:48.912749 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.912733    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-token\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:48.912819 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:48.912770    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-data\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:49.014301 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:49.014269    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nckmh\" (UniqueName: \"kubernetes.io/projected/71b1ff7d-a70f-45d8-8313-1f30c675f356-kube-api-access-nckmh\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:49.014496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:49.014344    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-workload-socket\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:49.014496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:49.014386    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-credential-socket\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:49.014496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:49.014414    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-podinfo\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:49.014496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:49.014437    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-envoy\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:49.014496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:49.014466    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-workload-certs\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:49.014781 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:49.014684    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-token\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:49.014781 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:49.014724    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-data\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:49.014781 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:49.014772    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b1ff7d-a70f-45d8-8313-1f30c675f356-istiod-ca-cert\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:49.014940 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:49.014774    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-workload-socket\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:49.014940 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:49.014843    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-credential-socket\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:49.015181 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:49.015154    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-workload-certs\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:49.015286 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:49.015226    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-data\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:49.015502 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:49.015481    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b1ff7d-a70f-45d8-8313-1f30c675f356-istiod-ca-cert\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:49.017270 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:49.017243    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-envoy\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:49.017595 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:49.017572    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-podinfo\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:49.036839 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:49.036810    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nckmh\" (UniqueName: \"kubernetes.io/projected/71b1ff7d-a70f-45d8-8313-1f30c675f356-kube-api-access-nckmh\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:49.036984 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:49.036811    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-token\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:49.042593 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:05:49.042556    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:05:49.149254 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:49.149175    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:49.485428 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:49.485397    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"]
Oct 30 01:05:49.487106 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:05:49.487064    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b1ff7d_a70f_45d8_8313_1f30c675f356.slice/crio-7a44d9d9cb457f74475f98c3a5920fa36b026118c3d28a1a4c4e230cf51f1d7c WatchSource:0}: Error finding container 7a44d9d9cb457f74475f98c3a5920fa36b026118c3d28a1a4c4e230cf51f1d7c: Status 404 returned error can't find the container with id 7a44d9d9cb457f74475f98c3a5920fa36b026118c3d28a1a4c4e230cf51f1d7c
Oct 30 01:05:49.489722 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:49.489690    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:05:49.489815 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:49.489755    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:05:49.489815 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:49.489785    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:05:50.115195 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:50.115148    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh" event={"ID":"71b1ff7d-a70f-45d8-8313-1f30c675f356","Type":"ContainerStarted","Data":"15dd57c06323771ce8ef84efbcc7aae4614bdf993d853fddd3fd289a15488373"}
Oct 30 01:05:50.115574 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:50.115249    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh" event={"ID":"71b1ff7d-a70f-45d8-8313-1f30c675f356","Type":"ContainerStarted","Data":"7a44d9d9cb457f74475f98c3a5920fa36b026118c3d28a1a4c4e230cf51f1d7c"}
Oct 30 01:05:50.149827 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:50.149796    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:50.151909 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:50.151879    4864 patch_prober.go:29] interesting pod/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 01:05:50.152107 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:50.151930    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh" podUID="71b1ff7d-a70f-45d8-8313-1f30c675f356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 01:05:50.187236 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:50.187181    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh" podStartSLOduration=2.187166512 podStartE2EDuration="2.187166512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:50.175776012 +0000 UTC m=+1462.513877927" watchObservedRunningTime="2026-10-30 01:05:50.187166512 +0000 UTC m=+1462.525268423"
Oct 30 01:05:50.243408 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:50.243373    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"]
Oct 30 01:05:50.243656 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:50.243627    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh" podUID="db5bae2a-6063-4cea-95ed-36df20be18ef" containerName="istio-proxy" containerID="cri-o://5fcb6b745a504bf0ed5fb8a1662d2c4474c5601d63060ca28eca9d54ef5e1142" gracePeriod=30
Oct 30 01:05:51.149634 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:51.149597    4864 patch_prober.go:29] interesting pod/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 01:05:51.150026 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:51.149652    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh" podUID="71b1ff7d-a70f-45d8-8313-1f30c675f356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 01:05:52.150233 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:52.150191    4864 patch_prober.go:29] interesting pod/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 01:05:52.150642 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:52.150252    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh" podUID="71b1ff7d-a70f-45d8-8313-1f30c675f356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 01:05:53.150375 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:53.150335    4864 patch_prober.go:29] interesting pod/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 01:05:53.150772 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:53.150402    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh" podUID="71b1ff7d-a70f-45d8-8313-1f30c675f356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 01:05:54.150026 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:54.149991    4864 patch_prober.go:29] interesting pod/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 01:05:54.150234 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:54.150044    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh" podUID="71b1ff7d-a70f-45d8-8313-1f30c675f356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 01:05:55.149791 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.149754    4864 patch_prober.go:29] interesting pod/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 01:05:55.150174 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.149816    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh" podUID="71b1ff7d-a70f-45d8-8313-1f30c675f356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 01:05:55.555049 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.555023    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:05:55.690242 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.690142    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db5bae2a-6063-4cea-95ed-36df20be18ef-kube-api-access-82f9r\" (UniqueName: \"kubernetes.io/projected/db5bae2a-6063-4cea-95ed-36df20be18ef-kube-api-access-82f9r\") pod \"db5bae2a-6063-4cea-95ed-36df20be18ef\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") "
Oct 30 01:05:55.690242 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.690204    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-workload-certs\") pod \"db5bae2a-6063-4cea-95ed-36df20be18ef\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") "
Oct 30 01:05:55.690454 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.690247    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-podinfo\") pod \"db5bae2a-6063-4cea-95ed-36df20be18ef\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") "
Oct 30 01:05:55.690454 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.690274    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db5bae2a-6063-4cea-95ed-36df20be18ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db5bae2a-6063-4cea-95ed-36df20be18ef-istiod-ca-cert\") pod \"db5bae2a-6063-4cea-95ed-36df20be18ef\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") "
Oct 30 01:05:55.690454 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.690313    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-token\" (UniqueName: \"kubernetes.io/projected/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-token\") pod \"db5bae2a-6063-4cea-95ed-36df20be18ef\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") "
Oct 30 01:05:55.690454 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.690347    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-data\") pod \"db5bae2a-6063-4cea-95ed-36df20be18ef\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") "
Oct 30 01:05:55.690454 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.690385    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-envoy\") pod \"db5bae2a-6063-4cea-95ed-36df20be18ef\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") "
Oct 30 01:05:55.690454 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.690433    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-credential-socket\") pod \"db5bae2a-6063-4cea-95ed-36df20be18ef\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") "
Oct 30 01:05:55.690773 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.690538    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-workload-socket\") pod \"db5bae2a-6063-4cea-95ed-36df20be18ef\" (UID: \"db5bae2a-6063-4cea-95ed-36df20be18ef\") "
Oct 30 01:05:55.690773 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.690572    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-workload-certs" pod "db5bae2a-6063-4cea-95ed-36df20be18ef" (UID: "db5bae2a-6063-4cea-95ed-36df20be18ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:55.690877 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.690752    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5bae2a-6063-4cea-95ed-36df20be18ef-istiod-ca-cert" pod "db5bae2a-6063-4cea-95ed-36df20be18ef" (UID: "db5bae2a-6063-4cea-95ed-36df20be18ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:55.690877 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.690857    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-credential-socket" pod "db5bae2a-6063-4cea-95ed-36df20be18ef" (UID: "db5bae2a-6063-4cea-95ed-36df20be18ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:55.690977 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.690838    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-data" pod "db5bae2a-6063-4cea-95ed-36df20be18ef" (UID: "db5bae2a-6063-4cea-95ed-36df20be18ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:55.691056 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.691029    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-workload-socket" pod "db5bae2a-6063-4cea-95ed-36df20be18ef" (UID: "db5bae2a-6063-4cea-95ed-36df20be18ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:55.691468 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.691441    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:55.691468 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.691468    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db5bae2a-6063-4cea-95ed-36df20be18ef-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:55.691652 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.691482    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:55.691652 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.691497    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:55.691652 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.691512    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:55.692743 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.692717    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db5bae2a-6063-4cea-95ed-36df20be18ef-kube-api-access-82f9r" pod "db5bae2a-6063-4cea-95ed-36df20be18ef" (UID: "db5bae2a-6063-4cea-95ed-36df20be18ef"). InnerVolumeSpecName "kube-api-access-82f9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:55.692906 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.692882    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-podinfo" pod "db5bae2a-6063-4cea-95ed-36df20be18ef" (UID: "db5bae2a-6063-4cea-95ed-36df20be18ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:05:55.693030 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.693000    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-envoy" pod "db5bae2a-6063-4cea-95ed-36df20be18ef" (UID: "db5bae2a-6063-4cea-95ed-36df20be18ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:55.693108 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.693012    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-token" pod "db5bae2a-6063-4cea-95ed-36df20be18ef" (UID: "db5bae2a-6063-4cea-95ed-36df20be18ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:55.792606 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.792571    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82f9r\" (UniqueName: \"kubernetes.io/projected/db5bae2a-6063-4cea-95ed-36df20be18ef-kube-api-access-82f9r\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:55.792606 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.792602    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:55.792842 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.792616    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:55.792842 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:55.792630    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db5bae2a-6063-4cea-95ed-36df20be18ef-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:05:56.154491 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:56.154461    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:56.154983 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:56.154962    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:56.155758 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:56.155737    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:05:56.158596 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:56.158571    4864 generic.go:356] "Generic (PLEG): container finished" podID="db5bae2a-6063-4cea-95ed-36df20be18ef" containerID="5fcb6b745a504bf0ed5fb8a1662d2c4474c5601d63060ca28eca9d54ef5e1142" exitCode=0
Oct 30 01:05:56.158682 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:56.158608    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"
Oct 30 01:05:56.158682 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:56.158645    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh" event={"ID":"db5bae2a-6063-4cea-95ed-36df20be18ef","Type":"ContainerDied","Data":"5fcb6b745a504bf0ed5fb8a1662d2c4474c5601d63060ca28eca9d54ef5e1142"}
Oct 30 01:05:56.158752 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:56.158691    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh" event={"ID":"db5bae2a-6063-4cea-95ed-36df20be18ef","Type":"ContainerDied","Data":"9cc9011c469aac74f881808baa1ea77a7f7792ee987be00bfa1518e3620952e8"}
Oct 30 01:05:56.158752 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:56.158702    4864 scope.go:122] "RemoveContainer" containerID="5fcb6b745a504bf0ed5fb8a1662d2c4474c5601d63060ca28eca9d54ef5e1142"
Oct 30 01:05:56.168811 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:56.168792    4864 scope.go:122] "RemoveContainer" containerID="f3a8dcea9ac3e886b0eb439175a271955563c53363326b2d0c2c6f3ef4a82bb6"
Oct 30 01:05:56.177977 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:56.177959    4864 scope.go:122] "RemoveContainer" containerID="5fcb6b745a504bf0ed5fb8a1662d2c4474c5601d63060ca28eca9d54ef5e1142"
Oct 30 01:05:56.178283 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:05:56.178232    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fcb6b745a504bf0ed5fb8a1662d2c4474c5601d63060ca28eca9d54ef5e1142\": container with ID starting with 5fcb6b745a504bf0ed5fb8a1662d2c4474c5601d63060ca28eca9d54ef5e1142 not found: ID does not exist" containerID="5fcb6b745a504bf0ed5fb8a1662d2c4474c5601d63060ca28eca9d54ef5e1142"
Oct 30 01:05:56.178369 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:56.178294    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fcb6b745a504bf0ed5fb8a1662d2c4474c5601d63060ca28eca9d54ef5e1142"} err="failed to get container status \"5fcb6b745a504bf0ed5fb8a1662d2c4474c5601d63060ca28eca9d54ef5e1142\": rpc error: code = NotFound desc = could not find container \"5fcb6b745a504bf0ed5fb8a1662d2c4474c5601d63060ca28eca9d54ef5e1142\": container with ID starting with 5fcb6b745a504bf0ed5fb8a1662d2c4474c5601d63060ca28eca9d54ef5e1142 not found: ID does not exist"
Oct 30 01:05:56.178369 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:56.178314    4864 scope.go:122] "RemoveContainer" containerID="f3a8dcea9ac3e886b0eb439175a271955563c53363326b2d0c2c6f3ef4a82bb6"
Oct 30 01:05:56.178588 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:05:56.178560    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a8dcea9ac3e886b0eb439175a271955563c53363326b2d0c2c6f3ef4a82bb6\": container with ID starting with f3a8dcea9ac3e886b0eb439175a271955563c53363326b2d0c2c6f3ef4a82bb6 not found: ID does not exist" containerID="f3a8dcea9ac3e886b0eb439175a271955563c53363326b2d0c2c6f3ef4a82bb6"
Oct 30 01:05:56.178637 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:56.178588    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a8dcea9ac3e886b0eb439175a271955563c53363326b2d0c2c6f3ef4a82bb6"} err="failed to get container status \"f3a8dcea9ac3e886b0eb439175a271955563c53363326b2d0c2c6f3ef4a82bb6\": rpc error: code = NotFound desc = could not find container \"f3a8dcea9ac3e886b0eb439175a271955563c53363326b2d0c2c6f3ef4a82bb6\": container with ID starting with f3a8dcea9ac3e886b0eb439175a271955563c53363326b2d0c2c6f3ef4a82bb6 not found: ID does not exist"
Oct 30 01:05:56.229996 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:56.229967    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"]
Oct 30 01:05:56.236606 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:56.236575    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1c78b8-openshift-default-67d995cbdf-t2gkh"]
Oct 30 01:05:57.796631 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:05:57.796593    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db5bae2a-6063-4cea-95ed-36df20be18ef" path="/var/lib/kubelet/pods/db5bae2a-6063-4cea-95ed-36df20be18ef/volumes"
Oct 30 01:06:19.044177 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:06:19.044060    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:06:27.986091 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:27.986054    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:06:27.987285 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:27.987268    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:06:34.154054 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:34.154019    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"]
Oct 30 01:06:34.154593 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:34.154268    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq" podUID="5e60eb51-e478-4cdc-b604-98e106f6bb4f" containerName="istio-proxy" containerID="cri-o://a11e7709cdf53c68bfa941109c6a1b550dd5a0c779fdd6b379b10ca4b774648c" gracePeriod=30
Oct 30 01:06:34.165549 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:34.165515    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"]
Oct 30 01:06:34.165739 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:34.165718    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh" podUID="71b1ff7d-a70f-45d8-8313-1f30c675f356" containerName="istio-proxy" containerID="cri-o://15dd57c06323771ce8ef84efbcc7aae4614bdf993d853fddd3fd289a15488373" gracePeriod=30
Oct 30 01:06:35.512291 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.512258    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"]
Oct 30 01:06:35.512996 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.512972    4864 cpu_manager.go:414] "containerMap: removing container" podUID="db5bae2a-6063-4cea-95ed-36df20be18ef" containerName="istio-proxy"
Oct 30 01:06:35.512996 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.512995    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db5bae2a-6063-4cea-95ed-36df20be18ef" containerName="istio-proxy"
Oct 30 01:06:35.513232 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.513021    4864 cpu_manager.go:414] "containerMap: removing container" podUID="e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" containerName="istio-proxy"
Oct 30 01:06:35.513232 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.513028    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e27f9baa-b6fc-4d5b-9c49-56e93c2a703b" containerName="istio-proxy"
Oct 30 01:06:35.513232 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.513155    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="db5bae2a-6063-4cea-95ed-36df20be18ef" containerName="istio-proxy"
Oct 30 01:06:35.513232 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.513168    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="db5bae2a-6063-4cea-95ed-36df20be18ef" containerName="istio-proxy"
Oct 30 01:06:35.518168 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.518150    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.531503 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.531480    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f05249-openshift-default-dockercfg-xjbh9\""
Oct 30 01:06:35.545747 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.545712    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"]
Oct 30 01:06:35.550173 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.550144    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a40d14-acf4-489f-bc5c-23449ea9c822-istiod-ca-cert\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.550308 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.550211    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-credential-socket\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.550308 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.550295    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-token\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.550398 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.550344    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-workload-socket\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.550455 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.550441    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-podinfo\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.550526 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.550511    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-data\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.550711 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.550688    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-envoy\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.550917 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.550894    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-workload-certs\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.551044 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.551019    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-799zp\" (UniqueName: \"kubernetes.io/projected/83a40d14-acf4-489f-bc5c-23449ea9c822-kube-api-access-799zp\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.668268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.665973    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-token\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.668268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.666051    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-workload-socket\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.668268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.666137    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-podinfo\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.668268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.666254    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-data\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.668268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.666312    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-envoy\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.668268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.666373    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-workload-certs\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.668268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.666457    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-799zp\" (UniqueName: \"kubernetes.io/projected/83a40d14-acf4-489f-bc5c-23449ea9c822-kube-api-access-799zp\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.668268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.666529    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a40d14-acf4-489f-bc5c-23449ea9c822-istiod-ca-cert\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.668268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.666575    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-credential-socket\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.668268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.667036    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-credential-socket\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.668268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.667653    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-workload-socket\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.669353 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.668374    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-workload-certs\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.669353 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.668603    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-data\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.672582 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.670367    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a40d14-acf4-489f-bc5c-23449ea9c822-istiod-ca-cert\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.682890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.682852    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-envoy\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.690105 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.690028    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-podinfo\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.709915 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.709883    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-token\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.712987 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.712914    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-799zp\" (UniqueName: \"kubernetes.io/projected/83a40d14-acf4-489f-bc5c-23449ea9c822-kube-api-access-799zp\") pod \"e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") " pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:35.835156 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:35.834905    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:36.176747 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:36.176716    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"]
Oct 30 01:06:36.179322 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:06:36.179287    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a40d14_acf4_489f_bc5c_23449ea9c822.slice/crio-7203fe472a3634fe5d1916a55118fe61ce884d938d21ff2cd44ecbb0a4136840 WatchSource:0}: Error finding container 7203fe472a3634fe5d1916a55118fe61ce884d938d21ff2cd44ecbb0a4136840: Status 404 returned error can't find the container with id 7203fe472a3634fe5d1916a55118fe61ce884d938d21ff2cd44ecbb0a4136840
Oct 30 01:06:36.182556 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:36.182519    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:06:36.182660 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:36.182606    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:06:36.182660 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:36.182650    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:06:36.436330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:36.436227    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht" event={"ID":"83a40d14-acf4-489f-bc5c-23449ea9c822","Type":"ContainerStarted","Data":"348da3d95ff31138dd2fccd825b2213e63ad6366c220d2e9a9ae6298ab76d559"}
Oct 30 01:06:36.436330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:36.436299    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht" event={"ID":"83a40d14-acf4-489f-bc5c-23449ea9c822","Type":"ContainerStarted","Data":"7203fe472a3634fe5d1916a55118fe61ce884d938d21ff2cd44ecbb0a4136840"}
Oct 30 01:06:36.494907 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:36.494847    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht" podStartSLOduration=1.494825015 podStartE2EDuration="1.494825015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:36.487946886 +0000 UTC m=+1508.826048796" watchObservedRunningTime="2026-10-30 01:06:36.494825015 +0000 UTC m=+1508.832926925"
Oct 30 01:06:36.835878 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:36.835838    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:37.842405 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:37.842371    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:38.449106 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:38.449066    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:38.450152 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:38.450122    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:06:39.439350 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.439312    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:06:39.457945 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.457914    4864 generic.go:356] "Generic (PLEG): container finished" podID="71b1ff7d-a70f-45d8-8313-1f30c675f356" containerID="15dd57c06323771ce8ef84efbcc7aae4614bdf993d853fddd3fd289a15488373" exitCode=0
Oct 30 01:06:39.458146 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.457982    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh" event={"ID":"71b1ff7d-a70f-45d8-8313-1f30c675f356","Type":"ContainerDied","Data":"15dd57c06323771ce8ef84efbcc7aae4614bdf993d853fddd3fd289a15488373"}
Oct 30 01:06:39.459581 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.459556    4864 generic.go:356] "Generic (PLEG): container finished" podID="5e60eb51-e478-4cdc-b604-98e106f6bb4f" containerID="a11e7709cdf53c68bfa941109c6a1b550dd5a0c779fdd6b379b10ca4b774648c" exitCode=0
Oct 30 01:06:39.459709 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.459622    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"
Oct 30 01:06:39.459709 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.459664    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq" event={"ID":"5e60eb51-e478-4cdc-b604-98e106f6bb4f","Type":"ContainerDied","Data":"a11e7709cdf53c68bfa941109c6a1b550dd5a0c779fdd6b379b10ca4b774648c"}
Oct 30 01:06:39.459821 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.459733    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq" event={"ID":"5e60eb51-e478-4cdc-b604-98e106f6bb4f","Type":"ContainerDied","Data":"d310e28d9502b5bdba2cd4b3e99d28ab301196afd026f272783efe08c86718f4"}
Oct 30 01:06:39.459821 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.459769    4864 scope.go:122] "RemoveContainer" containerID="a11e7709cdf53c68bfa941109c6a1b550dd5a0c779fdd6b379b10ca4b774648c"
Oct 30 01:06:39.470736 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.470707    4864 scope.go:122] "RemoveContainer" containerID="a11e7709cdf53c68bfa941109c6a1b550dd5a0c779fdd6b379b10ca4b774648c"
Oct 30 01:06:39.471209 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:06:39.471171    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a11e7709cdf53c68bfa941109c6a1b550dd5a0c779fdd6b379b10ca4b774648c\": container with ID starting with a11e7709cdf53c68bfa941109c6a1b550dd5a0c779fdd6b379b10ca4b774648c not found: ID does not exist" containerID="a11e7709cdf53c68bfa941109c6a1b550dd5a0c779fdd6b379b10ca4b774648c"
Oct 30 01:06:39.471312 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.471218    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a11e7709cdf53c68bfa941109c6a1b550dd5a0c779fdd6b379b10ca4b774648c"} err="failed to get container status \"a11e7709cdf53c68bfa941109c6a1b550dd5a0c779fdd6b379b10ca4b774648c\": rpc error: code = NotFound desc = could not find container \"a11e7709cdf53c68bfa941109c6a1b550dd5a0c779fdd6b379b10ca4b774648c\": container with ID starting with a11e7709cdf53c68bfa941109c6a1b550dd5a0c779fdd6b379b10ca4b774648c not found: ID does not exist"
Oct 30 01:06:39.500701 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.500668    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:06:39.540424 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.540378    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e60eb51-e478-4cdc-b604-98e106f6bb4f-kube-api-access-n5v2k\" (UniqueName: \"kubernetes.io/projected/5e60eb51-e478-4cdc-b604-98e106f6bb4f-kube-api-access-n5v2k\") pod \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") "
Oct 30 01:06:39.540635 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.540464    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-token\" (UniqueName: \"kubernetes.io/projected/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-token\") pod \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") "
Oct 30 01:06:39.540635 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.540614    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-workload-socket\") pod \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") "
Oct 30 01:06:39.540755 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.540729    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-data\") pod \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") "
Oct 30 01:06:39.540823 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.540756    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-workload-certs\") pod \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") "
Oct 30 01:06:39.540878 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.540845    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istiod-ca-cert\") pod \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") "
Oct 30 01:06:39.540966 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.540943    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-envoy\") pod \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") "
Oct 30 01:06:39.541061 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.541021    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-podinfo\") pod \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") "
Oct 30 01:06:39.541144 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.541058    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-credential-socket\") pod \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\" (UID: \"5e60eb51-e478-4cdc-b604-98e106f6bb4f\") "
Oct 30 01:06:39.544730 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.543803    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-credential-socket" pod "5e60eb51-e478-4cdc-b604-98e106f6bb4f" (UID: "5e60eb51-e478-4cdc-b604-98e106f6bb4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:39.544730 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.544354    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istiod-ca-cert" pod "5e60eb51-e478-4cdc-b604-98e106f6bb4f" (UID: "5e60eb51-e478-4cdc-b604-98e106f6bb4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:39.544730 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.544628    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-workload-certs" pod "5e60eb51-e478-4cdc-b604-98e106f6bb4f" (UID: "5e60eb51-e478-4cdc-b604-98e106f6bb4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:39.544730 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.544704    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-workload-socket" pod "5e60eb51-e478-4cdc-b604-98e106f6bb4f" (UID: "5e60eb51-e478-4cdc-b604-98e106f6bb4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:39.545342 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.545317    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-data" pod "5e60eb51-e478-4cdc-b604-98e106f6bb4f" (UID: "5e60eb51-e478-4cdc-b604-98e106f6bb4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:39.545447 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.545385    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-podinfo" pod "5e60eb51-e478-4cdc-b604-98e106f6bb4f" (UID: "5e60eb51-e478-4cdc-b604-98e106f6bb4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:06:39.545684 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.545657    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e60eb51-e478-4cdc-b604-98e106f6bb4f-kube-api-access-n5v2k" pod "5e60eb51-e478-4cdc-b604-98e106f6bb4f" (UID: "5e60eb51-e478-4cdc-b604-98e106f6bb4f"). InnerVolumeSpecName "kube-api-access-n5v2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:39.545861 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.545841    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-token" pod "5e60eb51-e478-4cdc-b604-98e106f6bb4f" (UID: "5e60eb51-e478-4cdc-b604-98e106f6bb4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:39.546668 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.546646    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-envoy" pod "5e60eb51-e478-4cdc-b604-98e106f6bb4f" (UID: "5e60eb51-e478-4cdc-b604-98e106f6bb4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:39.642478 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.642437    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-credential-socket\") pod \"71b1ff7d-a70f-45d8-8313-1f30c675f356\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") "
Oct 30 01:06:39.642674 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.642490    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b1ff7d-a70f-45d8-8313-1f30c675f356-kube-api-access-nckmh\" (UniqueName: \"kubernetes.io/projected/71b1ff7d-a70f-45d8-8313-1f30c675f356-kube-api-access-nckmh\") pod \"71b1ff7d-a70f-45d8-8313-1f30c675f356\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") "
Oct 30 01:06:39.642674 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.642533    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71b1ff7d-a70f-45d8-8313-1f30c675f356-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b1ff7d-a70f-45d8-8313-1f30c675f356-istiod-ca-cert\") pod \"71b1ff7d-a70f-45d8-8313-1f30c675f356\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") "
Oct 30 01:06:39.642674 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.642557    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-envoy\") pod \"71b1ff7d-a70f-45d8-8313-1f30c675f356\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") "
Oct 30 01:06:39.642674 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.642644    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-data\") pod \"71b1ff7d-a70f-45d8-8313-1f30c675f356\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") "
Oct 30 01:06:39.642674 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.642670    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-workload-socket\") pod \"71b1ff7d-a70f-45d8-8313-1f30c675f356\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") "
Oct 30 01:06:39.642934 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.642765    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-podinfo\") pod \"71b1ff7d-a70f-45d8-8313-1f30c675f356\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") "
Oct 30 01:06:39.642934 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.642762    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-credential-socket" pod "71b1ff7d-a70f-45d8-8313-1f30c675f356" (UID: "71b1ff7d-a70f-45d8-8313-1f30c675f356"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:39.643048 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.643014    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-workload-certs\") pod \"71b1ff7d-a70f-45d8-8313-1f30c675f356\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") "
Oct 30 01:06:39.643393 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.643116    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-data" pod "71b1ff7d-a70f-45d8-8313-1f30c675f356" (UID: "71b1ff7d-a70f-45d8-8313-1f30c675f356"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:39.643393 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.643156    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-workload-socket" pod "71b1ff7d-a70f-45d8-8313-1f30c675f356" (UID: "71b1ff7d-a70f-45d8-8313-1f30c675f356"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:39.643393 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.643208    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b1ff7d-a70f-45d8-8313-1f30c675f356-istiod-ca-cert" pod "71b1ff7d-a70f-45d8-8313-1f30c675f356" (UID: "71b1ff7d-a70f-45d8-8313-1f30c675f356"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:39.643393 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.643274    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-workload-certs" pod "71b1ff7d-a70f-45d8-8313-1f30c675f356" (UID: "71b1ff7d-a70f-45d8-8313-1f30c675f356"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:39.643393 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.643387    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-token\" (UniqueName: \"kubernetes.io/projected/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-token\") pod \"71b1ff7d-a70f-45d8-8313-1f30c675f356\" (UID: \"71b1ff7d-a70f-45d8-8313-1f30c675f356\") "
Oct 30 01:06:39.644062 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.644035    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.644062 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.644067    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.644263 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.644097    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.644263 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.644112    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.644263 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.644125    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.644263 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.644138    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.644263 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.644150    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.644263 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.644162    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b1ff7d-a70f-45d8-8313-1f30c675f356-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.644263 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.644177    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.644263 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.644190    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e60eb51-e478-4cdc-b604-98e106f6bb4f-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.644263 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.644204    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.644263 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.644216    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.644263 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.644230    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5v2k\" (UniqueName: \"kubernetes.io/projected/5e60eb51-e478-4cdc-b604-98e106f6bb4f-kube-api-access-n5v2k\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.644263 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.644244    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e60eb51-e478-4cdc-b604-98e106f6bb4f-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.645113 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.645086    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-envoy" pod "71b1ff7d-a70f-45d8-8313-1f30c675f356" (UID: "71b1ff7d-a70f-45d8-8313-1f30c675f356"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:39.645249 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.645225    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b1ff7d-a70f-45d8-8313-1f30c675f356-kube-api-access-nckmh" pod "71b1ff7d-a70f-45d8-8313-1f30c675f356" (UID: "71b1ff7d-a70f-45d8-8313-1f30c675f356"). InnerVolumeSpecName "kube-api-access-nckmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:39.645414 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.645395    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-token" pod "71b1ff7d-a70f-45d8-8313-1f30c675f356" (UID: "71b1ff7d-a70f-45d8-8313-1f30c675f356"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:39.645450 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.645396    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-podinfo" pod "71b1ff7d-a70f-45d8-8313-1f30c675f356" (UID: "71b1ff7d-a70f-45d8-8313-1f30c675f356"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:06:39.745233 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.745203    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.745233 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.745228    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.745233 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.745239    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nckmh\" (UniqueName: \"kubernetes.io/projected/71b1ff7d-a70f-45d8-8313-1f30c675f356-kube-api-access-nckmh\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.745475 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.745250    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b1ff7d-a70f-45d8-8313-1f30c675f356-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.805574 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.805542    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"]
Oct 30 01:06:39.809700 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:39.809673    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-mzpqq"]
Oct 30 01:06:40.489185 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:40.488584    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"
Oct 30 01:06:40.489688 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:40.489217    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh" event={"ID":"71b1ff7d-a70f-45d8-8313-1f30c675f356","Type":"ContainerDied","Data":"7a44d9d9cb457f74475f98c3a5920fa36b026118c3d28a1a4c4e230cf51f1d7c"}
Oct 30 01:06:40.489688 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:40.489270    4864 scope.go:122] "RemoveContainer" containerID="15dd57c06323771ce8ef84efbcc7aae4614bdf993d853fddd3fd289a15488373"
Oct 30 01:06:40.538180 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:40.538149    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"]
Oct 30 01:06:40.548119 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:40.548066    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-qcjhh"]
Oct 30 01:06:41.793729 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:41.793695    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e60eb51-e478-4cdc-b604-98e106f6bb4f" path="/var/lib/kubelet/pods/5e60eb51-e478-4cdc-b604-98e106f6bb4f/volumes"
Oct 30 01:06:41.794247 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:06:41.794230    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71b1ff7d-a70f-45d8-8313-1f30c675f356" path="/var/lib/kubelet/pods/71b1ff7d-a70f-45d8-8313-1f30c675f356/volumes"
Oct 30 01:06:49.042135 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:06:49.042091    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:07:19.044889 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:07:19.044853    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:07:20.476346 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:20.476310    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"]
Oct 30 01:07:20.476910 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:20.476894    4864 cpu_manager.go:414] "containerMap: removing container" podUID="db5bae2a-6063-4cea-95ed-36df20be18ef" containerName="istio-proxy"
Oct 30 01:07:20.476957 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:20.476911    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db5bae2a-6063-4cea-95ed-36df20be18ef" containerName="istio-proxy"
Oct 30 01:07:20.476957 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:20.476923    4864 cpu_manager.go:414] "containerMap: removing container" podUID="71b1ff7d-a70f-45d8-8313-1f30c675f356" containerName="istio-proxy"
Oct 30 01:07:20.476957 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:20.476928    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b1ff7d-a70f-45d8-8313-1f30c675f356" containerName="istio-proxy"
Oct 30 01:07:20.477055 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:20.476968    4864 cpu_manager.go:414] "containerMap: removing container" podUID="5e60eb51-e478-4cdc-b604-98e106f6bb4f" containerName="istio-proxy"
Oct 30 01:07:20.477055 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:20.476974    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e60eb51-e478-4cdc-b604-98e106f6bb4f" containerName="istio-proxy"
Oct 30 01:07:20.477141 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:20.477058    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e60eb51-e478-4cdc-b604-98e106f6bb4f" containerName="istio-proxy"
Oct 30 01:07:20.477141 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:20.477083    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b1ff7d-a70f-45d8-8313-1f30c675f356" containerName="istio-proxy"
Oct 30 01:07:20.481768 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:20.481748    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"
Oct 30 01:07:20.499293 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:20.499263    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f05249-serving-cert\""
Oct 30 01:07:20.514255 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:20.514221    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"]
Oct 30 01:07:20.649229 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:20.649190    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0967d271-ad67-457d-bd9a-de6522a24f55-metrics-certs\") pod \"maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx\" (UID: \"0967d271-ad67-457d-bd9a-de6522a24f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"
Oct 30 01:07:20.649400 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:20.649319    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0967d271-ad67-457d-bd9a-de6522a24f55-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx\" (UID: \"0967d271-ad67-457d-bd9a-de6522a24f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"
Oct 30 01:07:20.649479 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:20.649462    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0967d271-ad67-457d-bd9a-de6522a24f55-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx\" (UID: \"0967d271-ad67-457d-bd9a-de6522a24f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"
Oct 30 01:07:20.649521 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:20.649504    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld9rp\" (UniqueName: \"kubernetes.io/projected/0967d271-ad67-457d-bd9a-de6522a24f55-kube-api-access-ld9rp\") pod \"maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx\" (UID: \"0967d271-ad67-457d-bd9a-de6522a24f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"
Oct 30 01:07:20.751111 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:20.750998    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0967d271-ad67-457d-bd9a-de6522a24f55-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx\" (UID: \"0967d271-ad67-457d-bd9a-de6522a24f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"
Oct 30 01:07:20.751273 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:20.751141    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld9rp\" (UniqueName: \"kubernetes.io/projected/0967d271-ad67-457d-bd9a-de6522a24f55-kube-api-access-ld9rp\") pod \"maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx\" (UID: \"0967d271-ad67-457d-bd9a-de6522a24f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"
Oct 30 01:07:20.751320 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:20.751305    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0967d271-ad67-457d-bd9a-de6522a24f55-metrics-certs\") pod \"maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx\" (UID: \"0967d271-ad67-457d-bd9a-de6522a24f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"
Oct 30 01:07:20.751421 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:20.751405    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0967d271-ad67-457d-bd9a-de6522a24f55-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx\" (UID: \"0967d271-ad67-457d-bd9a-de6522a24f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"
Oct 30 01:07:20.751612 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:07:20.751586    4864 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-serving-cert: secret "maas-api-e2e-ipp-a-f05249-serving-cert" not found
Oct 30 01:07:20.751716 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:07:20.751675    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0967d271-ad67-457d-bd9a-de6522a24f55-maas-api-tls podName:0967d271-ad67-457d-bd9a-de6522a24f55 nodeName:}" failed. No retries permitted until 2026-10-30 01:07:21.251655692 +0000 UTC m=+1553.589757595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0967d271-ad67-457d-bd9a-de6522a24f55-maas-api-tls") pod "maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx" (UID: "0967d271-ad67-457d-bd9a-de6522a24f55") : secret "maas-api-e2e-ipp-a-f05249-serving-cert" not found
Oct 30 01:07:20.751773 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:20.751736    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0967d271-ad67-457d-bd9a-de6522a24f55-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx\" (UID: \"0967d271-ad67-457d-bd9a-de6522a24f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"
Oct 30 01:07:20.754005 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:20.753975    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0967d271-ad67-457d-bd9a-de6522a24f55-metrics-certs\") pod \"maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx\" (UID: \"0967d271-ad67-457d-bd9a-de6522a24f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"
Oct 30 01:07:20.777249 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:20.777218    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld9rp\" (UniqueName: \"kubernetes.io/projected/0967d271-ad67-457d-bd9a-de6522a24f55-kube-api-access-ld9rp\") pod \"maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx\" (UID: \"0967d271-ad67-457d-bd9a-de6522a24f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"
Oct 30 01:07:21.258045 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:21.258010    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0967d271-ad67-457d-bd9a-de6522a24f55-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx\" (UID: \"0967d271-ad67-457d-bd9a-de6522a24f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"
Oct 30 01:07:21.260860 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:21.260834    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0967d271-ad67-457d-bd9a-de6522a24f55-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx\" (UID: \"0967d271-ad67-457d-bd9a-de6522a24f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"
Oct 30 01:07:21.399865 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:21.399784    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"
Oct 30 01:07:21.717433 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:21.717402    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"]
Oct 30 01:07:21.718733 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:07:21.718704    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0967d271_ad67_457d_bd9a_de6522a24f55.slice/crio-c0ba1f39c4a4d81b2f589c2d1db6394e18f339481bfee7ba1738bb0238dc952c WatchSource:0}: Error finding container c0ba1f39c4a4d81b2f589c2d1db6394e18f339481bfee7ba1738bb0238dc952c: Status 404 returned error can't find the container with id c0ba1f39c4a4d81b2f589c2d1db6394e18f339481bfee7ba1738bb0238dc952c
Oct 30 01:07:21.720170 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:21.720152    4864 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:07:21.766527 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:21.766496    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx" event={"ID":"0967d271-ad67-457d-bd9a-de6522a24f55","Type":"ContainerStarted","Data":"c0ba1f39c4a4d81b2f589c2d1db6394e18f339481bfee7ba1738bb0238dc952c"}
Oct 30 01:07:22.773878 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:22.773841    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx" event={"ID":"0967d271-ad67-457d-bd9a-de6522a24f55","Type":"ContainerStarted","Data":"ef93133e32a7b3ac05fb6fa124f1634bdfec5da10a658cfdcc40674b6b2a78ff"}
Oct 30 01:07:22.774345 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:22.773975    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"
Oct 30 01:07:22.812872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:22.812815    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx" podStartSLOduration=2.367366129 podStartE2EDuration="2.812800344s" totalImagesPullingTime="445.434215ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:22.812258236 +0000 UTC m=+1555.150360143" watchObservedRunningTime="2026-10-30 01:07:22.812800344 +0000 UTC m=+1555.150902253"
Oct 30 01:07:28.785699 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:28.785670    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"
Oct 30 01:07:40.695684 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:40.695591    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"]
Oct 30 01:07:40.701518 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:40.701495    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:40.718910 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:40.718813    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b43bff-openshift-default-dockercfg-w4wqv\""
Oct 30 01:07:40.735016 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:40.734968    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"]
Oct 30 01:07:40.815264 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:40.815219    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-workload-certs\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:40.815461 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:40.815377    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30401bdf-ea14-4729-9507-f81eb6685a43-istio-podinfo\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:40.815575 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:40.815553    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-istio-envoy\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:40.815668 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:40.815634    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30401bdf-ea14-4729-9507-f81eb6685a43-istio-token\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:40.815668 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:40.815662    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7bdd\" (UniqueName: \"kubernetes.io/projected/30401bdf-ea14-4729-9507-f81eb6685a43-kube-api-access-t7bdd\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:40.815828 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:40.815711    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-workload-socket\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:40.815828 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:40.815743    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-credential-socket\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:40.815828 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:40.815783    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30401bdf-ea14-4729-9507-f81eb6685a43-istiod-ca-cert\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:40.816066 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:40.815856    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-istio-data\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:40.925099 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:40.916864    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30401bdf-ea14-4729-9507-f81eb6685a43-istio-podinfo\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:41.018322 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:41.018282    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-istio-envoy\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:41.023172 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:41.018389    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30401bdf-ea14-4729-9507-f81eb6685a43-istio-token\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:41.023172 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:41.018417    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7bdd\" (UniqueName: \"kubernetes.io/projected/30401bdf-ea14-4729-9507-f81eb6685a43-kube-api-access-t7bdd\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:41.023172 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:41.018456    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-workload-socket\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:41.023172 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:41.018479    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-credential-socket\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:41.023172 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:41.018511    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30401bdf-ea14-4729-9507-f81eb6685a43-istiod-ca-cert\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:41.023172 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:41.018563    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-istio-data\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:41.023172 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:41.018637    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-workload-certs\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:41.023172 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:40.967054    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30401bdf-ea14-4729-9507-f81eb6685a43-istio-podinfo\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:41.023172 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:41.019242    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-workload-certs\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:41.023172 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:41.019505    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-workload-socket\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:41.023172 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:41.019737    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-credential-socket\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:41.023172 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:41.020369    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30401bdf-ea14-4729-9507-f81eb6685a43-istiod-ca-cert\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:41.023172 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:41.020903    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-istio-data\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:41.023172 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:41.021410    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-istio-envoy\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:41.045307 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:41.045276    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30401bdf-ea14-4729-9507-f81eb6685a43-istio-token\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:41.046017 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:41.045991    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7bdd\" (UniqueName: \"kubernetes.io/projected/30401bdf-ea14-4729-9507-f81eb6685a43-kube-api-access-t7bdd\") pod \"e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") " pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:41.325101 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:41.325002    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:41.692389 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:41.692344    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"]
Oct 30 01:07:41.693607 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:07:41.693561    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30401bdf_ea14_4729_9507_f81eb6685a43.slice/crio-708fe00a7ad4412af9dc2f484c83dc6b3e16823503fe029165830c47f0bb29f8 WatchSource:0}: Error finding container 708fe00a7ad4412af9dc2f484c83dc6b3e16823503fe029165830c47f0bb29f8: Status 404 returned error can't find the container with id 708fe00a7ad4412af9dc2f484c83dc6b3e16823503fe029165830c47f0bb29f8
Oct 30 01:07:41.696157 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:41.696116    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:07:41.696507 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:41.696210    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:07:41.696507 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:41.696242    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:07:42.030739 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:42.030702    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc" event={"ID":"30401bdf-ea14-4729-9507-f81eb6685a43","Type":"ContainerStarted","Data":"b8ad9a337f2f2203a98968b09860c9632ad9bed41d6e4b94fd981efd255c418a"}
Oct 30 01:07:42.030983 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:42.030754    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc" event={"ID":"30401bdf-ea14-4729-9507-f81eb6685a43","Type":"ContainerStarted","Data":"708fe00a7ad4412af9dc2f484c83dc6b3e16823503fe029165830c47f0bb29f8"}
Oct 30 01:07:42.096219 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:42.096166    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc" podStartSLOduration=2.096151987 podStartE2EDuration="2.096151987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:42.085800515 +0000 UTC m=+1574.423902425" watchObservedRunningTime="2026-10-30 01:07:42.096151987 +0000 UTC m=+1574.434253897"
Oct 30 01:07:42.325495 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:42.325408    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:42.331032 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:42.331004    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:43.041019 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:43.040986    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:43.046585 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:43.046526    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:07:48.083372 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:48.083329    4864 generic.go:356] "Generic (PLEG): container finished" podID="83a40d14-acf4-489f-bc5c-23449ea9c822" containerID="348da3d95ff31138dd2fccd825b2213e63ad6366c220d2e9a9ae6298ab76d559" exitCode=0
Oct 30 01:07:48.083912 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:48.083398    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht" event={"ID":"83a40d14-acf4-489f-bc5c-23449ea9c822","Type":"ContainerDied","Data":"348da3d95ff31138dd2fccd825b2213e63ad6366c220d2e9a9ae6298ab76d559"}
Oct 30 01:07:48.083999 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:48.083979    4864 scope.go:122] "RemoveContainer" containerID="348da3d95ff31138dd2fccd825b2213e63ad6366c220d2e9a9ae6298ab76d559"
Oct 30 01:07:48.085654 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:48.085618    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:07:48.085771 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:48.085691    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:07:48.085771 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:48.085726    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:07:49.041053 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:07:49.041018    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:07:49.092211 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:49.092176    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht" event={"ID":"83a40d14-acf4-489f-bc5c-23449ea9c822","Type":"ContainerStarted","Data":"a683ce1fa2d2db92e7977dec3cdb193b9d4ad6821ef0a94b9ae71f6eb77df0a8"}
Oct 30 01:07:49.836653 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:49.836612    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:07:49.838140 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:49.838110    4864 patch_prober.go:29] interesting pod/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 01:07:49.838283 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:49.838172    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht" podUID="83a40d14-acf4-489f-bc5c-23449ea9c822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 01:07:50.836801 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:50.836764    4864 patch_prober.go:29] interesting pod/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 01:07:50.837227 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:50.836835    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht" podUID="83a40d14-acf4-489f-bc5c-23449ea9c822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 01:07:51.835851 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:51.835815    4864 patch_prober.go:29] interesting pod/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 01:07:51.836032 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:51.835874    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht" podUID="83a40d14-acf4-489f-bc5c-23449ea9c822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 01:07:52.836367 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:52.836333    4864 patch_prober.go:29] interesting pod/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 01:07:52.836771 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:52.836398    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht" podUID="83a40d14-acf4-489f-bc5c-23449ea9c822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 01:07:53.449562 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:53.449527    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:07:53.838676 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:53.838583    4864 patch_prober.go:29] interesting pod/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 01:07:53.839368 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:53.839315    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht" podUID="83a40d14-acf4-489f-bc5c-23449ea9c822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 01:07:54.840848 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:54.840815    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:07:54.847491 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:07:54.844923    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:08:19.043257 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:08:19.043209    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:08:49.041096 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:08:49.041030    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:08:49.757232 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:08:49.757198    4864 generic.go:356] "Generic (PLEG): container finished" podID="30401bdf-ea14-4729-9507-f81eb6685a43" containerID="b8ad9a337f2f2203a98968b09860c9632ad9bed41d6e4b94fd981efd255c418a" exitCode=0
Oct 30 01:08:49.757444 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:08:49.757276    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc" event={"ID":"30401bdf-ea14-4729-9507-f81eb6685a43","Type":"ContainerDied","Data":"b8ad9a337f2f2203a98968b09860c9632ad9bed41d6e4b94fd981efd255c418a"}
Oct 30 01:08:49.757763 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:08:49.757747    4864 scope.go:122] "RemoveContainer" containerID="b8ad9a337f2f2203a98968b09860c9632ad9bed41d6e4b94fd981efd255c418a"
Oct 30 01:08:49.759456 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:08:49.759429    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:08:49.759527 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:08:49.759497    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:08:49.759572 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:08:49.759531    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:08:50.785377 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:08:50.785339    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc" event={"ID":"30401bdf-ea14-4729-9507-f81eb6685a43","Type":"ContainerStarted","Data":"10ade92dab0a0f334af25e795f46c144596fe47377e9c388b569f3038cbe78ec"}
Oct 30 01:08:51.325663 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:08:51.325629    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:08:51.325946 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:08:51.325923    4864 patch_prober.go:29] interesting pod/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 01:08:51.326040 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:08:51.325974    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc" podUID="30401bdf-ea14-4729-9507-f81eb6685a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 01:08:52.326458 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:08:52.326419    4864 patch_prober.go:29] interesting pod/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 01:08:52.326879 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:08:52.326490    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc" podUID="30401bdf-ea14-4729-9507-f81eb6685a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 01:08:53.325482 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:08:53.325445    4864 patch_prober.go:29] interesting pod/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 01:08:53.325671 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:08:53.325500    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc" podUID="30401bdf-ea14-4729-9507-f81eb6685a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 01:08:54.326420 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:08:54.326383    4864 patch_prober.go:29] interesting pod/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 01:08:54.326830 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:08:54.326438    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc" podUID="30401bdf-ea14-4729-9507-f81eb6685a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 01:08:55.325583 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:08:55.325546    4864 patch_prober.go:29] interesting pod/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 01:08:55.325806 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:08:55.325600    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc" podUID="30401bdf-ea14-4729-9507-f81eb6685a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 01:08:56.330452 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:08:56.330422    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:08:56.330924 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:08:56.330902    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:08:56.331661 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:08:56.331638    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:09:14.010153 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.010055    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"]
Oct 30 01:09:14.014589 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.014570    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.037377 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.037349    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-4ec143-openshift-default-dockercfg-g7lhg\""
Oct 30 01:09:14.053049 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.053014    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"]
Oct 30 01:09:14.192743 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.192710    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w92wp\" (UniqueName: \"kubernetes.io/projected/79304a2e-fd29-4e09-acd9-4d727ae11a48-kube-api-access-w92wp\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.192919 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.192768    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-workload-certs\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.192962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.192916    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-credential-socket\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.193068 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.193049    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-workload-socket\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.193186 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.193165    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-data\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.193223 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.193205    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-podinfo\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.193293 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.193272    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-envoy\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.193339 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.193302    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-token\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.193339 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.193329    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79304a2e-fd29-4e09-acd9-4d727ae11a48-istiod-ca-cert\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.295041 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.294945    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-workload-socket\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.295369 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.295347    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-data\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.295496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.295479    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-podinfo\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.295637 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.295623    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-envoy\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.295736 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.295722    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-token\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.295851 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.295838    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79304a2e-fd29-4e09-acd9-4d727ae11a48-istiod-ca-cert\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.296005 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.295991    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w92wp\" (UniqueName: \"kubernetes.io/projected/79304a2e-fd29-4e09-acd9-4d727ae11a48-kube-api-access-w92wp\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.296181 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.296154    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-workload-certs\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.296347 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.296330    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-credential-socket\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.297015 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.296977    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-credential-socket\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.299108 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.299065    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79304a2e-fd29-4e09-acd9-4d727ae11a48-istiod-ca-cert\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.304351 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.299881    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-workload-socket\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.304509 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.304008    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-workload-certs\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.304604 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.304293    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-data\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.322705 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.322674    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-podinfo\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.323105 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.323083    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-envoy\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.348178 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.333579    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w92wp\" (UniqueName: \"kubernetes.io/projected/79304a2e-fd29-4e09-acd9-4d727ae11a48-kube-api-access-w92wp\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.348178 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.337678    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-token\") pod \"e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") " pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.629327 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.629234    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:14.978162 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.978139    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"]
Oct 30 01:09:14.980813 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:09:14.980787    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79304a2e_fd29_4e09_acd9_4d727ae11a48.slice/crio-315d3398e18c22a9017d677a7d0072a9d6ba6e2a5415b32c44e2229fd4a94323 WatchSource:0}: Error finding container 315d3398e18c22a9017d677a7d0072a9d6ba6e2a5415b32c44e2229fd4a94323: Status 404 returned error can't find the container with id 315d3398e18c22a9017d677a7d0072a9d6ba6e2a5415b32c44e2229fd4a94323
Oct 30 01:09:14.983485 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.983454    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:09:14.983558 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.983521    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:09:14.983593 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:14.983556    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:09:15.953283 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:15.953247    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx" event={"ID":"79304a2e-fd29-4e09-acd9-4d727ae11a48","Type":"ContainerStarted","Data":"51c022560da22957b55edcc20c0f4f7d9bff806cce51ed570c805f486e1ec998"}
Oct 30 01:09:15.953709 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:15.953306    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx" event={"ID":"79304a2e-fd29-4e09-acd9-4d727ae11a48","Type":"ContainerStarted","Data":"315d3398e18c22a9017d677a7d0072a9d6ba6e2a5415b32c44e2229fd4a94323"}
Oct 30 01:09:16.006439 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:16.006378    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx" podStartSLOduration=3.006361884 podStartE2EDuration="3.006361884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:16.006157833 +0000 UTC m=+1668.344259744" watchObservedRunningTime="2026-10-30 01:09:16.006361884 +0000 UTC m=+1668.344463797"
Oct 30 01:09:16.629679 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:16.629601    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:16.644113 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:16.642181    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:16.959661 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:16.959635    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:16.960589 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:16.960568    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:09:19.041326 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:09:19.041287    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:09:49.040978 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:09:49.040939    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:09:53.882589 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:53.882554    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"]
Oct 30 01:09:53.883131 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:53.882807    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n" podUID="67f83fb2-248a-49c8-ba87-ee314ff0006e" containerName="istio-proxy" containerID="cri-o://2d930fd4af2e9d74b91b410e8662e7fa006307b800e8034c7a50e0ff90adf6df" gracePeriod=30
Oct 30 01:09:59.207409 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.207362    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 01:09:59.319309 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.319276    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67f83fb2-248a-49c8-ba87-ee314ff0006e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f83fb2-248a-49c8-ba87-ee314ff0006e-istiod-ca-cert\") pod \"67f83fb2-248a-49c8-ba87-ee314ff0006e\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") "
Oct 30 01:09:59.319504 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.319391    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67f83fb2-248a-49c8-ba87-ee314ff0006e-kube-api-access-dbt7b\" (UniqueName: \"kubernetes.io/projected/67f83fb2-248a-49c8-ba87-ee314ff0006e-kube-api-access-dbt7b\") pod \"67f83fb2-248a-49c8-ba87-ee314ff0006e\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") "
Oct 30 01:09:59.319504 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.319429    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-token\" (UniqueName: \"kubernetes.io/projected/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-token\") pod \"67f83fb2-248a-49c8-ba87-ee314ff0006e\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") "
Oct 30 01:09:59.319615 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.319543    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-workload-socket\") pod \"67f83fb2-248a-49c8-ba87-ee314ff0006e\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") "
Oct 30 01:09:59.319666 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.319654    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-data\") pod \"67f83fb2-248a-49c8-ba87-ee314ff0006e\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") "
Oct 30 01:09:59.319720 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.319708    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-envoy\") pod \"67f83fb2-248a-49c8-ba87-ee314ff0006e\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") "
Oct 30 01:09:59.319776 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.319746    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-podinfo\") pod \"67f83fb2-248a-49c8-ba87-ee314ff0006e\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") "
Oct 30 01:09:59.319831 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.319788    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-workload-certs\") pod \"67f83fb2-248a-49c8-ba87-ee314ff0006e\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") "
Oct 30 01:09:59.319831 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.319699    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f83fb2-248a-49c8-ba87-ee314ff0006e-istiod-ca-cert" pod "67f83fb2-248a-49c8-ba87-ee314ff0006e" (UID: "67f83fb2-248a-49c8-ba87-ee314ff0006e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:59.319929 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.319838    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-credential-socket\") pod \"67f83fb2-248a-49c8-ba87-ee314ff0006e\" (UID: \"67f83fb2-248a-49c8-ba87-ee314ff0006e\") "
Oct 30 01:09:59.319929 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.319915    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-workload-socket" pod "67f83fb2-248a-49c8-ba87-ee314ff0006e" (UID: "67f83fb2-248a-49c8-ba87-ee314ff0006e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:59.320276 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.320231    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-data" pod "67f83fb2-248a-49c8-ba87-ee314ff0006e" (UID: "67f83fb2-248a-49c8-ba87-ee314ff0006e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:59.320497 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.320303    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-workload-certs" pod "67f83fb2-248a-49c8-ba87-ee314ff0006e" (UID: "67f83fb2-248a-49c8-ba87-ee314ff0006e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:59.323020 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.320582    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-credential-socket" pod "67f83fb2-248a-49c8-ba87-ee314ff0006e" (UID: "67f83fb2-248a-49c8-ba87-ee314ff0006e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:59.323020 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.320690    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.323020 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.320716    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.323020 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.320734    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.323020 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.320747    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f83fb2-248a-49c8-ba87-ee314ff0006e-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.323020 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.322222    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f83fb2-248a-49c8-ba87-ee314ff0006e-kube-api-access-dbt7b" pod "67f83fb2-248a-49c8-ba87-ee314ff0006e" (UID: "67f83fb2-248a-49c8-ba87-ee314ff0006e"). InnerVolumeSpecName "kube-api-access-dbt7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:59.323020 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.322265    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-envoy" pod "67f83fb2-248a-49c8-ba87-ee314ff0006e" (UID: "67f83fb2-248a-49c8-ba87-ee314ff0006e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:59.323020 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.322285    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-podinfo" pod "67f83fb2-248a-49c8-ba87-ee314ff0006e" (UID: "67f83fb2-248a-49c8-ba87-ee314ff0006e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:09:59.323020 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.322393    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-token" pod "67f83fb2-248a-49c8-ba87-ee314ff0006e" (UID: "67f83fb2-248a-49c8-ba87-ee314ff0006e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:59.324804 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.324776    4864 generic.go:356] "Generic (PLEG): container finished" podID="67f83fb2-248a-49c8-ba87-ee314ff0006e" containerID="2d930fd4af2e9d74b91b410e8662e7fa006307b800e8034c7a50e0ff90adf6df" exitCode=0
Oct 30 01:09:59.324916 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.324817    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"
Oct 30 01:09:59.324916 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.324856    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n" event={"ID":"67f83fb2-248a-49c8-ba87-ee314ff0006e","Type":"ContainerDied","Data":"2d930fd4af2e9d74b91b410e8662e7fa006307b800e8034c7a50e0ff90adf6df"}
Oct 30 01:09:59.325028 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.324921    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n" event={"ID":"67f83fb2-248a-49c8-ba87-ee314ff0006e","Type":"ContainerDied","Data":"634e86b3b47092eeb9a7e06723c3907bddcbb95994118c77dbb590405612475f"}
Oct 30 01:09:59.325028 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.324955    4864 scope.go:122] "RemoveContainer" containerID="2d930fd4af2e9d74b91b410e8662e7fa006307b800e8034c7a50e0ff90adf6df"
Oct 30 01:09:59.351789 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.351766    4864 scope.go:122] "RemoveContainer" containerID="2d930fd4af2e9d74b91b410e8662e7fa006307b800e8034c7a50e0ff90adf6df"
Oct 30 01:09:59.352149 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:09:59.352122    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d930fd4af2e9d74b91b410e8662e7fa006307b800e8034c7a50e0ff90adf6df\": container with ID starting with 2d930fd4af2e9d74b91b410e8662e7fa006307b800e8034c7a50e0ff90adf6df not found: ID does not exist" containerID="2d930fd4af2e9d74b91b410e8662e7fa006307b800e8034c7a50e0ff90adf6df"
Oct 30 01:09:59.352209 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.352163    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d930fd4af2e9d74b91b410e8662e7fa006307b800e8034c7a50e0ff90adf6df"} err="failed to get container status \"2d930fd4af2e9d74b91b410e8662e7fa006307b800e8034c7a50e0ff90adf6df\": rpc error: code = NotFound desc = could not find container \"2d930fd4af2e9d74b91b410e8662e7fa006307b800e8034c7a50e0ff90adf6df\": container with ID starting with 2d930fd4af2e9d74b91b410e8662e7fa006307b800e8034c7a50e0ff90adf6df not found: ID does not exist"
Oct 30 01:09:59.386149 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.386117    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"]
Oct 30 01:09:59.392773 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.392743    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-47h6n"]
Oct 30 01:09:59.421923 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.421848    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbt7b\" (UniqueName: \"kubernetes.io/projected/67f83fb2-248a-49c8-ba87-ee314ff0006e-kube-api-access-dbt7b\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.421923 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.421875    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.421923 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.421888    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.421923 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.421896    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f83fb2-248a-49c8-ba87-ee314ff0006e-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.421923 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.421906    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f83fb2-248a-49c8-ba87-ee314ff0006e-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.795110 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:09:59.795064    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67f83fb2-248a-49c8-ba87-ee314ff0006e" path="/var/lib/kubelet/pods/67f83fb2-248a-49c8-ba87-ee314ff0006e/volumes"
Oct 30 01:10:05.552388 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:05.552355    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"]
Oct 30 01:10:05.552874 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:05.552586    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt" podUID="7c965c38-598d-47ce-9104-5a0b85c6a925" containerName="istio-proxy" containerID="cri-o://169eef1a050b2df2b8fc45d99406a76467632798d355bdf7db38a6e7acb30a5e" gracePeriod=30
Oct 30 01:10:08.925251 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:08.925217    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"]
Oct 30 01:10:08.925672 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:08.925436    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl" podUID="c136a701-915b-409a-8b28-198deed55a38" containerName="istio-proxy" containerID="cri-o://4b1110e22b0a9ed98daadffa08d6094d3affbe74ba27c2923dc8913565f3bb2b" gracePeriod=30
Oct 30 01:10:10.074224 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:10.074190    4864 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:10:10.074601 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:10.074250    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt" podUID="7c965c38-598d-47ce-9104-5a0b85c6a925" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:10:10.921044 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:10.921010    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 01:10:11.003948 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.003916    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-workload-socket\") pod \"7c965c38-598d-47ce-9104-5a0b85c6a925\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") "
Oct 30 01:10:11.004157 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.003973    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-credential-socket\") pod \"7c965c38-598d-47ce-9104-5a0b85c6a925\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") "
Oct 30 01:10:11.004157 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.003993    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c965c38-598d-47ce-9104-5a0b85c6a925-kube-api-access-27hrx\" (UniqueName: \"kubernetes.io/projected/7c965c38-598d-47ce-9104-5a0b85c6a925-kube-api-access-27hrx\") pod \"7c965c38-598d-47ce-9104-5a0b85c6a925\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") "
Oct 30 01:10:11.004157 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.004036    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-podinfo\") pod \"7c965c38-598d-47ce-9104-5a0b85c6a925\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") "
Oct 30 01:10:11.004326 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.004184    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-token\" (UniqueName: \"kubernetes.io/projected/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-token\") pod \"7c965c38-598d-47ce-9104-5a0b85c6a925\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") "
Oct 30 01:10:11.004326 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.004262    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-workload-socket" pod "7c965c38-598d-47ce-9104-5a0b85c6a925" (UID: "7c965c38-598d-47ce-9104-5a0b85c6a925"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:11.004326 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.004280    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-credential-socket" pod "7c965c38-598d-47ce-9104-5a0b85c6a925" (UID: "7c965c38-598d-47ce-9104-5a0b85c6a925"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:11.004326 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.004322    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-envoy\") pod \"7c965c38-598d-47ce-9104-5a0b85c6a925\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") "
Oct 30 01:10:11.004515 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.004418    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c965c38-598d-47ce-9104-5a0b85c6a925-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c965c38-598d-47ce-9104-5a0b85c6a925-istiod-ca-cert\") pod \"7c965c38-598d-47ce-9104-5a0b85c6a925\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") "
Oct 30 01:10:11.004515 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.004483    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-workload-certs\") pod \"7c965c38-598d-47ce-9104-5a0b85c6a925\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") "
Oct 30 01:10:11.004619 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.004525    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-data\") pod \"7c965c38-598d-47ce-9104-5a0b85c6a925\" (UID: \"7c965c38-598d-47ce-9104-5a0b85c6a925\") "
Oct 30 01:10:11.005062 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.005010    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-workload-certs" pod "7c965c38-598d-47ce-9104-5a0b85c6a925" (UID: "7c965c38-598d-47ce-9104-5a0b85c6a925"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:11.005062 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.005030    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c965c38-598d-47ce-9104-5a0b85c6a925-istiod-ca-cert" pod "7c965c38-598d-47ce-9104-5a0b85c6a925" (UID: "7c965c38-598d-47ce-9104-5a0b85c6a925"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:11.005387 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.005230    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-data" pod "7c965c38-598d-47ce-9104-5a0b85c6a925" (UID: "7c965c38-598d-47ce-9104-5a0b85c6a925"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:11.005387 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.005297    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c965c38-598d-47ce-9104-5a0b85c6a925-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:11.005387 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.005312    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:11.005387 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.005321    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:11.005387 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.005330    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:11.007409 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.007383    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-envoy" pod "7c965c38-598d-47ce-9104-5a0b85c6a925" (UID: "7c965c38-598d-47ce-9104-5a0b85c6a925"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:11.010454 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.010429    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-token" pod "7c965c38-598d-47ce-9104-5a0b85c6a925" (UID: "7c965c38-598d-47ce-9104-5a0b85c6a925"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:11.010570 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.010464    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c965c38-598d-47ce-9104-5a0b85c6a925-kube-api-access-27hrx" pod "7c965c38-598d-47ce-9104-5a0b85c6a925" (UID: "7c965c38-598d-47ce-9104-5a0b85c6a925"). InnerVolumeSpecName "kube-api-access-27hrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:11.011760 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.011732    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-podinfo" pod "7c965c38-598d-47ce-9104-5a0b85c6a925" (UID: "7c965c38-598d-47ce-9104-5a0b85c6a925"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:10:11.107085 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.107045    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:11.107462 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.107099    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:11.107462 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.107109    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:11.107462 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.107118    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27hrx\" (UniqueName: \"kubernetes.io/projected/7c965c38-598d-47ce-9104-5a0b85c6a925-kube-api-access-27hrx\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:11.107462 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.107127    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c965c38-598d-47ce-9104-5a0b85c6a925-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:11.443791 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.443751    4864 generic.go:356] "Generic (PLEG): container finished" podID="7c965c38-598d-47ce-9104-5a0b85c6a925" containerID="169eef1a050b2df2b8fc45d99406a76467632798d355bdf7db38a6e7acb30a5e" exitCode=0
Oct 30 01:10:11.444129 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.444106    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt" event={"ID":"7c965c38-598d-47ce-9104-5a0b85c6a925","Type":"ContainerDied","Data":"169eef1a050b2df2b8fc45d99406a76467632798d355bdf7db38a6e7acb30a5e"}
Oct 30 01:10:11.444331 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.444314    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt" event={"ID":"7c965c38-598d-47ce-9104-5a0b85c6a925","Type":"ContainerDied","Data":"fae9f4ef2b1537c585318e22c37385b6c92a97016e55e7752175726d6889773e"}
Oct 30 01:10:11.444453 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.444440    4864 scope.go:122] "RemoveContainer" containerID="169eef1a050b2df2b8fc45d99406a76467632798d355bdf7db38a6e7acb30a5e"
Oct 30 01:10:11.444731 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.444714    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"
Oct 30 01:10:11.478318 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.478283    4864 scope.go:122] "RemoveContainer" containerID="169eef1a050b2df2b8fc45d99406a76467632798d355bdf7db38a6e7acb30a5e"
Oct 30 01:10:11.478733 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:10:11.478703    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"169eef1a050b2df2b8fc45d99406a76467632798d355bdf7db38a6e7acb30a5e\": container with ID starting with 169eef1a050b2df2b8fc45d99406a76467632798d355bdf7db38a6e7acb30a5e not found: ID does not exist" containerID="169eef1a050b2df2b8fc45d99406a76467632798d355bdf7db38a6e7acb30a5e"
Oct 30 01:10:11.478849 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.478748    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"169eef1a050b2df2b8fc45d99406a76467632798d355bdf7db38a6e7acb30a5e"} err="failed to get container status \"169eef1a050b2df2b8fc45d99406a76467632798d355bdf7db38a6e7acb30a5e\": rpc error: code = NotFound desc = could not find container \"169eef1a050b2df2b8fc45d99406a76467632798d355bdf7db38a6e7acb30a5e\": container with ID starting with 169eef1a050b2df2b8fc45d99406a76467632798d355bdf7db38a6e7acb30a5e not found: ID does not exist"
Oct 30 01:10:11.505481 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.505443    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"]
Oct 30 01:10:11.508653 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.508628    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-xjhtt"]
Oct 30 01:10:11.799643 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:11.799605    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c965c38-598d-47ce-9104-5a0b85c6a925" path="/var/lib/kubelet/pods/7c965c38-598d-47ce-9104-5a0b85c6a925/volumes"
Oct 30 01:10:14.226278 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.226250    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 01:10:14.250811 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.250777    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-workload-socket\") pod \"c136a701-915b-409a-8b28-198deed55a38\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") "
Oct 30 01:10:14.250811 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.250814    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-istio-envoy\") pod \"c136a701-915b-409a-8b28-198deed55a38\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") "
Oct 30 01:10:14.251025 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.250869    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-credential-socket\") pod \"c136a701-915b-409a-8b28-198deed55a38\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") "
Oct 30 01:10:14.251025 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.250917    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-istio-data\") pod \"c136a701-915b-409a-8b28-198deed55a38\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") "
Oct 30 01:10:14.251025 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.250975    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c136a701-915b-409a-8b28-198deed55a38-kube-api-access-9rbp6\" (UniqueName: \"kubernetes.io/projected/c136a701-915b-409a-8b28-198deed55a38-kube-api-access-9rbp6\") pod \"c136a701-915b-409a-8b28-198deed55a38\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") "
Oct 30 01:10:14.251210 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.251034    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c136a701-915b-409a-8b28-198deed55a38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c136a701-915b-409a-8b28-198deed55a38-istiod-ca-cert\") pod \"c136a701-915b-409a-8b28-198deed55a38\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") "
Oct 30 01:10:14.251210 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.251058    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-workload-socket" pod "c136a701-915b-409a-8b28-198deed55a38" (UID: "c136a701-915b-409a-8b28-198deed55a38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:14.251210 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.251102    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-workload-certs\") pod \"c136a701-915b-409a-8b28-198deed55a38\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") "
Oct 30 01:10:14.251361 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.251231    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-credential-socket" pod "c136a701-915b-409a-8b28-198deed55a38" (UID: "c136a701-915b-409a-8b28-198deed55a38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:14.251361 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.251286    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c136a701-915b-409a-8b28-198deed55a38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c136a701-915b-409a-8b28-198deed55a38-istio-podinfo\") pod \"c136a701-915b-409a-8b28-198deed55a38\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") "
Oct 30 01:10:14.251361 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.251291    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-istio-data" pod "c136a701-915b-409a-8b28-198deed55a38" (UID: "c136a701-915b-409a-8b28-198deed55a38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:14.251361 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.251326    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c136a701-915b-409a-8b28-198deed55a38-istio-token\" (UniqueName: \"kubernetes.io/projected/c136a701-915b-409a-8b28-198deed55a38-istio-token\") pod \"c136a701-915b-409a-8b28-198deed55a38\" (UID: \"c136a701-915b-409a-8b28-198deed55a38\") "
Oct 30 01:10:14.251361 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.251344    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-workload-certs" pod "c136a701-915b-409a-8b28-198deed55a38" (UID: "c136a701-915b-409a-8b28-198deed55a38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:14.251567 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.251497    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c136a701-915b-409a-8b28-198deed55a38-istiod-ca-cert" pod "c136a701-915b-409a-8b28-198deed55a38" (UID: "c136a701-915b-409a-8b28-198deed55a38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:14.251996 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.251970    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:14.252149 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.251999    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:14.252149 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.252017    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c136a701-915b-409a-8b28-198deed55a38-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:14.252149 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.252031    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:14.252149 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.252042    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:14.253240 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.253212    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-istio-envoy" pod "c136a701-915b-409a-8b28-198deed55a38" (UID: "c136a701-915b-409a-8b28-198deed55a38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:14.253352 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.253236    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c136a701-915b-409a-8b28-198deed55a38-istio-podinfo" pod "c136a701-915b-409a-8b28-198deed55a38" (UID: "c136a701-915b-409a-8b28-198deed55a38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:10:14.253352 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.253265    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c136a701-915b-409a-8b28-198deed55a38-kube-api-access-9rbp6" pod "c136a701-915b-409a-8b28-198deed55a38" (UID: "c136a701-915b-409a-8b28-198deed55a38"). InnerVolumeSpecName "kube-api-access-9rbp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:14.253430 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.253379    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c136a701-915b-409a-8b28-198deed55a38-istio-token" pod "c136a701-915b-409a-8b28-198deed55a38" (UID: "c136a701-915b-409a-8b28-198deed55a38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:14.352847 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.352806    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c136a701-915b-409a-8b28-198deed55a38-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:14.352847 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.352841    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c136a701-915b-409a-8b28-198deed55a38-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:14.352847 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.352851    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c136a701-915b-409a-8b28-198deed55a38-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:14.353109 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.352860    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rbp6\" (UniqueName: \"kubernetes.io/projected/c136a701-915b-409a-8b28-198deed55a38-kube-api-access-9rbp6\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:14.489131 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.488118    4864 generic.go:356] "Generic (PLEG): container finished" podID="c136a701-915b-409a-8b28-198deed55a38" containerID="4b1110e22b0a9ed98daadffa08d6094d3affbe74ba27c2923dc8913565f3bb2b" exitCode=0
Oct 30 01:10:14.489131 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.488261    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl" event={"ID":"c136a701-915b-409a-8b28-198deed55a38","Type":"ContainerDied","Data":"4b1110e22b0a9ed98daadffa08d6094d3affbe74ba27c2923dc8913565f3bb2b"}
Oct 30 01:10:14.489131 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.488304    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl" event={"ID":"c136a701-915b-409a-8b28-198deed55a38","Type":"ContainerDied","Data":"f097284fe643a4ceaa723ad1f14553ba87df15619733e2a2d9814896d2e0fadf"}
Oct 30 01:10:14.489131 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.488325    4864 scope.go:122] "RemoveContainer" containerID="4b1110e22b0a9ed98daadffa08d6094d3affbe74ba27c2923dc8913565f3bb2b"
Oct 30 01:10:14.489131 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.488417    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"
Oct 30 01:10:14.500433 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.500411    4864 scope.go:122] "RemoveContainer" containerID="4b1110e22b0a9ed98daadffa08d6094d3affbe74ba27c2923dc8913565f3bb2b"
Oct 30 01:10:14.500715 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:10:14.500688    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b1110e22b0a9ed98daadffa08d6094d3affbe74ba27c2923dc8913565f3bb2b\": container with ID starting with 4b1110e22b0a9ed98daadffa08d6094d3affbe74ba27c2923dc8913565f3bb2b not found: ID does not exist" containerID="4b1110e22b0a9ed98daadffa08d6094d3affbe74ba27c2923dc8913565f3bb2b"
Oct 30 01:10:14.500774 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.500728    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b1110e22b0a9ed98daadffa08d6094d3affbe74ba27c2923dc8913565f3bb2b"} err="failed to get container status \"4b1110e22b0a9ed98daadffa08d6094d3affbe74ba27c2923dc8913565f3bb2b\": rpc error: code = NotFound desc = could not find container \"4b1110e22b0a9ed98daadffa08d6094d3affbe74ba27c2923dc8913565f3bb2b\": container with ID starting with 4b1110e22b0a9ed98daadffa08d6094d3affbe74ba27c2923dc8913565f3bb2b not found: ID does not exist"
Oct 30 01:10:14.537049 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.537022    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"]
Oct 30 01:10:14.540714 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:14.540687    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-pj6tl"]
Oct 30 01:10:15.794805 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:15.794762    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c136a701-915b-409a-8b28-198deed55a38" path="/var/lib/kubelet/pods/c136a701-915b-409a-8b28-198deed55a38/volumes"
Oct 30 01:10:16.088126 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:16.088013    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"]
Oct 30 01:10:16.088316 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:16.088280    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw" podUID="370c4c48-476a-45b1-bb4b-09f1d92d0fa1" containerName="istio-proxy" containerID="cri-o://fa4f560e8b55bcd008e187a3600d54b82f0e1cb7582a6f4381c0c1b96b0dd6e2" gracePeriod=30
Oct 30 01:10:17.417327 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:17.417295    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6k58k_d2a355c3-0703-422f-8ddb-555c4a364ade/payload-processing/0.log"
Oct 30 01:10:17.623942 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:17.623910    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6k58k_d2a355c3-0703-422f-8ddb-555c4a364ade/payload-processing/0.log"
Oct 30 01:10:19.046335 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:10:19.046297    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:10:21.375305 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.375279    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:10:21.495348 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.495313    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-podinfo\") pod \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") "
Oct 30 01:10:21.495718 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.495694    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-envoy\") pod \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") "
Oct 30 01:10:21.495929 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.495912    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-workload-certs\") pod \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") "
Oct 30 01:10:21.496045 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.496030    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-credential-socket\") pod \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") "
Oct 30 01:10:21.496188 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.496174    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istiod-ca-cert\") pod \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") "
Oct 30 01:10:21.496332 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.496318    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-data\") pod \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") "
Oct 30 01:10:21.496475 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.496461    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-workload-socket\") pod \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") "
Oct 30 01:10:21.496774 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.496751    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-token\" (UniqueName: \"kubernetes.io/projected/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-token\") pod \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") "
Oct 30 01:10:21.496953 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.496936    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-kube-api-access-lpcst\" (UniqueName: \"kubernetes.io/projected/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-kube-api-access-lpcst\") pod \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\" (UID: \"370c4c48-476a-45b1-bb4b-09f1d92d0fa1\") "
Oct 30 01:10:21.498105 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.497617    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-data" pod "370c4c48-476a-45b1-bb4b-09f1d92d0fa1" (UID: "370c4c48-476a-45b1-bb4b-09f1d92d0fa1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:21.498105 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.497680    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-workload-socket" pod "370c4c48-476a-45b1-bb4b-09f1d92d0fa1" (UID: "370c4c48-476a-45b1-bb4b-09f1d92d0fa1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:21.498105 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.497702    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-workload-certs" pod "370c4c48-476a-45b1-bb4b-09f1d92d0fa1" (UID: "370c4c48-476a-45b1-bb4b-09f1d92d0fa1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:21.498105 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.497910    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-credential-socket" pod "370c4c48-476a-45b1-bb4b-09f1d92d0fa1" (UID: "370c4c48-476a-45b1-bb4b-09f1d92d0fa1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:21.498984 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.498962    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:21.499129 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.499113    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:21.499237 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.499213    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:21.499237 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.499235    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:21.500782 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.500756    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istiod-ca-cert" pod "370c4c48-476a-45b1-bb4b-09f1d92d0fa1" (UID: "370c4c48-476a-45b1-bb4b-09f1d92d0fa1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:21.501498 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.501470    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-kube-api-access-lpcst" pod "370c4c48-476a-45b1-bb4b-09f1d92d0fa1" (UID: "370c4c48-476a-45b1-bb4b-09f1d92d0fa1"). InnerVolumeSpecName "kube-api-access-lpcst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:21.503275 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.503160    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-podinfo" pod "370c4c48-476a-45b1-bb4b-09f1d92d0fa1" (UID: "370c4c48-476a-45b1-bb4b-09f1d92d0fa1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:10:21.503275 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.503254    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-token" pod "370c4c48-476a-45b1-bb4b-09f1d92d0fa1" (UID: "370c4c48-476a-45b1-bb4b-09f1d92d0fa1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:21.505685 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.505649    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-envoy" pod "370c4c48-476a-45b1-bb4b-09f1d92d0fa1" (UID: "370c4c48-476a-45b1-bb4b-09f1d92d0fa1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:21.556961 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.556925    4864 generic.go:356] "Generic (PLEG): container finished" podID="370c4c48-476a-45b1-bb4b-09f1d92d0fa1" containerID="fa4f560e8b55bcd008e187a3600d54b82f0e1cb7582a6f4381c0c1b96b0dd6e2" exitCode=0
Oct 30 01:10:21.557180 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.556971    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"
Oct 30 01:10:21.557180 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.557012    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw" event={"ID":"370c4c48-476a-45b1-bb4b-09f1d92d0fa1","Type":"ContainerDied","Data":"fa4f560e8b55bcd008e187a3600d54b82f0e1cb7582a6f4381c0c1b96b0dd6e2"}
Oct 30 01:10:21.557180 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.557098    4864 scope.go:122] "RemoveContainer" containerID="fa4f560e8b55bcd008e187a3600d54b82f0e1cb7582a6f4381c0c1b96b0dd6e2"
Oct 30 01:10:21.557334 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.557098    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw" event={"ID":"370c4c48-476a-45b1-bb4b-09f1d92d0fa1","Type":"ContainerDied","Data":"b8aea7345cf557a3e0b117fc8c65ab4e60a6b0a2d5e1ceb0f795175892b7c5ba"}
Oct 30 01:10:21.572104 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.571950    4864 scope.go:122] "RemoveContainer" containerID="fa4f560e8b55bcd008e187a3600d54b82f0e1cb7582a6f4381c0c1b96b0dd6e2"
Oct 30 01:10:21.573905 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:10:21.573828    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa4f560e8b55bcd008e187a3600d54b82f0e1cb7582a6f4381c0c1b96b0dd6e2\": container with ID starting with fa4f560e8b55bcd008e187a3600d54b82f0e1cb7582a6f4381c0c1b96b0dd6e2 not found: ID does not exist" containerID="fa4f560e8b55bcd008e187a3600d54b82f0e1cb7582a6f4381c0c1b96b0dd6e2"
Oct 30 01:10:21.573905 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.573871    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa4f560e8b55bcd008e187a3600d54b82f0e1cb7582a6f4381c0c1b96b0dd6e2"} err="failed to get container status \"fa4f560e8b55bcd008e187a3600d54b82f0e1cb7582a6f4381c0c1b96b0dd6e2\": rpc error: code = NotFound desc = could not find container \"fa4f560e8b55bcd008e187a3600d54b82f0e1cb7582a6f4381c0c1b96b0dd6e2\": container with ID starting with fa4f560e8b55bcd008e187a3600d54b82f0e1cb7582a6f4381c0c1b96b0dd6e2 not found: ID does not exist"
Oct 30 01:10:21.600496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.600427    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:21.600496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.600454    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpcst\" (UniqueName: \"kubernetes.io/projected/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-kube-api-access-lpcst\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:21.600496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.600463    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:21.600496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.600472    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:21.600496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.600480    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370c4c48-476a-45b1-bb4b-09f1d92d0fa1-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:21.616346 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.616316    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"]
Oct 30 01:10:21.625634 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.625598    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-grnvw"]
Oct 30 01:10:21.796533 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:21.796498    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="370c4c48-476a-45b1-bb4b-09f1d92d0fa1" path="/var/lib/kubelet/pods/370c4c48-476a-45b1-bb4b-09f1d92d0fa1/volumes"
Oct 30 01:10:22.564544 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:22.564514    4864 generic.go:356] "Generic (PLEG): container finished" podID="79304a2e-fd29-4e09-acd9-4d727ae11a48" containerID="51c022560da22957b55edcc20c0f4f7d9bff806cce51ed570c805f486e1ec998" exitCode=0
Oct 30 01:10:22.564953 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:22.564592    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx" event={"ID":"79304a2e-fd29-4e09-acd9-4d727ae11a48","Type":"ContainerDied","Data":"51c022560da22957b55edcc20c0f4f7d9bff806cce51ed570c805f486e1ec998"}
Oct 30 01:10:22.565066 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:22.565046    4864 scope.go:122] "RemoveContainer" containerID="51c022560da22957b55edcc20c0f4f7d9bff806cce51ed570c805f486e1ec998"
Oct 30 01:10:22.566723 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:22.566697    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:10:22.566819 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:22.566763    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:10:22.566819 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:22.566795    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:10:23.600876 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:23.600836    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx" event={"ID":"79304a2e-fd29-4e09-acd9-4d727ae11a48","Type":"ContainerStarted","Data":"2466e28668c8ed5ac454badcc15187aa24bdadb5df29732f321e5ad14f38c7d3"}
Oct 30 01:10:23.630110 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:23.630080    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:10:23.630306 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:23.630256    4864 patch_prober.go:29] interesting pod/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 01:10:23.630357 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:23.630340    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx" podUID="79304a2e-fd29-4e09-acd9-4d727ae11a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 01:10:24.630407 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:24.630370    4864 patch_prober.go:29] interesting pod/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 01:10:24.630825 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:24.630434    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx" podUID="79304a2e-fd29-4e09-acd9-4d727ae11a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 01:10:25.629750 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:25.629718    4864 patch_prober.go:29] interesting pod/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 01:10:25.629921 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:25.629769    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx" podUID="79304a2e-fd29-4e09-acd9-4d727ae11a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 01:10:26.630262 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:26.630226    4864 patch_prober.go:29] interesting pod/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 01:10:26.630767 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:26.630285    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx" podUID="79304a2e-fd29-4e09-acd9-4d727ae11a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 01:10:27.631814 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:27.631782    4864 patch_prober.go:29] interesting pod/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 01:10:27.632443 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:27.632410    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx" podUID="79304a2e-fd29-4e09-acd9-4d727ae11a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 01:10:28.630113 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:28.630061    4864 patch_prober.go:29] interesting pod/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 01:10:28.630296 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:28.630145    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx" podUID="79304a2e-fd29-4e09-acd9-4d727ae11a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 01:10:28.889144 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:28.884024    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"]
Oct 30 01:10:28.889144 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:28.884326    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc" podUID="30401bdf-ea14-4729-9507-f81eb6685a43" containerName="istio-proxy" containerID="cri-o://10ade92dab0a0f334af25e795f46c144596fe47377e9c388b569f3038cbe78ec" gracePeriod=30
Oct 30 01:10:29.634820 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:29.634790    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:10:29.635351 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:29.635327    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:10:29.636142 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:29.636123    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:10:29.907751 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:29.907622    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"]
Oct 30 01:10:29.908393 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:29.907879    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d" podUID="b460b90d-0d55-44af-a4b9-2e70b3a7b679" containerName="istio-proxy" containerID="cri-o://65b1a10ab81c4ae278a3dbb13c7a38f6ac885ce43be98ada7c28696af5455ad3" gracePeriod=30
Oct 30 01:10:34.064734 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.064706    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:10:34.132971 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.132933    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-credential-socket\") pod \"30401bdf-ea14-4729-9507-f81eb6685a43\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") "
Oct 30 01:10:34.133165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.133065    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-istio-envoy\") pod \"30401bdf-ea14-4729-9507-f81eb6685a43\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") "
Oct 30 01:10:34.133165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.133125    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30401bdf-ea14-4729-9507-f81eb6685a43-istio-token\" (UniqueName: \"kubernetes.io/projected/30401bdf-ea14-4729-9507-f81eb6685a43-istio-token\") pod \"30401bdf-ea14-4729-9507-f81eb6685a43\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") "
Oct 30 01:10:34.133288 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.133164    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-workload-socket\") pod \"30401bdf-ea14-4729-9507-f81eb6685a43\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") "
Oct 30 01:10:34.133288 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.133231    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-istio-data\") pod \"30401bdf-ea14-4729-9507-f81eb6685a43\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") "
Oct 30 01:10:34.133288 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.133270    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30401bdf-ea14-4729-9507-f81eb6685a43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30401bdf-ea14-4729-9507-f81eb6685a43-istiod-ca-cert\") pod \"30401bdf-ea14-4729-9507-f81eb6685a43\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") "
Oct 30 01:10:34.133438 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.133362    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30401bdf-ea14-4729-9507-f81eb6685a43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30401bdf-ea14-4729-9507-f81eb6685a43-istio-podinfo\") pod \"30401bdf-ea14-4729-9507-f81eb6685a43\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") "
Oct 30 01:10:34.133438 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.133431    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-workload-certs\") pod \"30401bdf-ea14-4729-9507-f81eb6685a43\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") "
Oct 30 01:10:34.133544 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.133476    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30401bdf-ea14-4729-9507-f81eb6685a43-kube-api-access-t7bdd\" (UniqueName: \"kubernetes.io/projected/30401bdf-ea14-4729-9507-f81eb6685a43-kube-api-access-t7bdd\") pod \"30401bdf-ea14-4729-9507-f81eb6685a43\" (UID: \"30401bdf-ea14-4729-9507-f81eb6685a43\") "
Oct 30 01:10:34.133722 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.133691    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-istio-data" pod "30401bdf-ea14-4729-9507-f81eb6685a43" (UID: "30401bdf-ea14-4729-9507-f81eb6685a43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:34.133790 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.133759    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30401bdf-ea14-4729-9507-f81eb6685a43-istiod-ca-cert" pod "30401bdf-ea14-4729-9507-f81eb6685a43" (UID: "30401bdf-ea14-4729-9507-f81eb6685a43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:34.134109 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.134085    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:34.134227 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.134113    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30401bdf-ea14-4729-9507-f81eb6685a43-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:34.134384 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.134358    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-workload-certs" pod "30401bdf-ea14-4729-9507-f81eb6685a43" (UID: "30401bdf-ea14-4729-9507-f81eb6685a43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:34.134509 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.134431    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-workload-socket" pod "30401bdf-ea14-4729-9507-f81eb6685a43" (UID: "30401bdf-ea14-4729-9507-f81eb6685a43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:34.134509 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.134482    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-credential-socket" pod "30401bdf-ea14-4729-9507-f81eb6685a43" (UID: "30401bdf-ea14-4729-9507-f81eb6685a43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:34.136374 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.136349    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30401bdf-ea14-4729-9507-f81eb6685a43-kube-api-access-t7bdd" pod "30401bdf-ea14-4729-9507-f81eb6685a43" (UID: "30401bdf-ea14-4729-9507-f81eb6685a43"). InnerVolumeSpecName "kube-api-access-t7bdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:34.136476 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.136455    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30401bdf-ea14-4729-9507-f81eb6685a43-istio-token" pod "30401bdf-ea14-4729-9507-f81eb6685a43" (UID: "30401bdf-ea14-4729-9507-f81eb6685a43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:34.136532 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.136474    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30401bdf-ea14-4729-9507-f81eb6685a43-istio-podinfo" pod "30401bdf-ea14-4729-9507-f81eb6685a43" (UID: "30401bdf-ea14-4729-9507-f81eb6685a43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:10:34.136532 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.136518    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-istio-envoy" pod "30401bdf-ea14-4729-9507-f81eb6685a43" (UID: "30401bdf-ea14-4729-9507-f81eb6685a43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:34.244816 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.244676    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:34.244816 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.244712    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30401bdf-ea14-4729-9507-f81eb6685a43-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:34.244816 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.244728    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:34.244816 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.244741    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30401bdf-ea14-4729-9507-f81eb6685a43-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:34.244816 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.244757    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:34.244816 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.244771    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7bdd\" (UniqueName: \"kubernetes.io/projected/30401bdf-ea14-4729-9507-f81eb6685a43-kube-api-access-t7bdd\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:34.244816 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.244785    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30401bdf-ea14-4729-9507-f81eb6685a43-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:34.542949 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.542911    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:10:34.543148 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.542991    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d" podUID="b460b90d-0d55-44af-a4b9-2e70b3a7b679" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:10:34.752096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.750574    4864 generic.go:356] "Generic (PLEG): container finished" podID="30401bdf-ea14-4729-9507-f81eb6685a43" containerID="10ade92dab0a0f334af25e795f46c144596fe47377e9c388b569f3038cbe78ec" exitCode=0
Oct 30 01:10:34.752096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.750713    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc" event={"ID":"30401bdf-ea14-4729-9507-f81eb6685a43","Type":"ContainerDied","Data":"10ade92dab0a0f334af25e795f46c144596fe47377e9c388b569f3038cbe78ec"}
Oct 30 01:10:34.752096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.750752    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc" event={"ID":"30401bdf-ea14-4729-9507-f81eb6685a43","Type":"ContainerDied","Data":"708fe00a7ad4412af9dc2f484c83dc6b3e16823503fe029165830c47f0bb29f8"}
Oct 30 01:10:34.752096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.750771    4864 scope.go:122] "RemoveContainer" containerID="10ade92dab0a0f334af25e795f46c144596fe47377e9c388b569f3038cbe78ec"
Oct 30 01:10:34.752096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.750902    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"
Oct 30 01:10:34.771614 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.771590    4864 scope.go:122] "RemoveContainer" containerID="b8ad9a337f2f2203a98968b09860c9632ad9bed41d6e4b94fd981efd255c418a"
Oct 30 01:10:34.780848 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.780827    4864 scope.go:122] "RemoveContainer" containerID="10ade92dab0a0f334af25e795f46c144596fe47377e9c388b569f3038cbe78ec"
Oct 30 01:10:34.781158 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:10:34.781139    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10ade92dab0a0f334af25e795f46c144596fe47377e9c388b569f3038cbe78ec\": container with ID starting with 10ade92dab0a0f334af25e795f46c144596fe47377e9c388b569f3038cbe78ec not found: ID does not exist" containerID="10ade92dab0a0f334af25e795f46c144596fe47377e9c388b569f3038cbe78ec"
Oct 30 01:10:34.781216 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.781168    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ade92dab0a0f334af25e795f46c144596fe47377e9c388b569f3038cbe78ec"} err="failed to get container status \"10ade92dab0a0f334af25e795f46c144596fe47377e9c388b569f3038cbe78ec\": rpc error: code = NotFound desc = could not find container \"10ade92dab0a0f334af25e795f46c144596fe47377e9c388b569f3038cbe78ec\": container with ID starting with 10ade92dab0a0f334af25e795f46c144596fe47377e9c388b569f3038cbe78ec not found: ID does not exist"
Oct 30 01:10:34.781216 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.781184    4864 scope.go:122] "RemoveContainer" containerID="b8ad9a337f2f2203a98968b09860c9632ad9bed41d6e4b94fd981efd255c418a"
Oct 30 01:10:34.781410 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:10:34.781394    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8ad9a337f2f2203a98968b09860c9632ad9bed41d6e4b94fd981efd255c418a\": container with ID starting with b8ad9a337f2f2203a98968b09860c9632ad9bed41d6e4b94fd981efd255c418a not found: ID does not exist" containerID="b8ad9a337f2f2203a98968b09860c9632ad9bed41d6e4b94fd981efd255c418a"
Oct 30 01:10:34.781450 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.781416    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ad9a337f2f2203a98968b09860c9632ad9bed41d6e4b94fd981efd255c418a"} err="failed to get container status \"b8ad9a337f2f2203a98968b09860c9632ad9bed41d6e4b94fd981efd255c418a\": rpc error: code = NotFound desc = could not find container \"b8ad9a337f2f2203a98968b09860c9632ad9bed41d6e4b94fd981efd255c418a\": container with ID starting with b8ad9a337f2f2203a98968b09860c9632ad9bed41d6e4b94fd981efd255c418a not found: ID does not exist"
Oct 30 01:10:34.801964 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.801887    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"]
Oct 30 01:10:34.806319 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:34.806292    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b43bff-openshift-default-6548cfb8d9-4s5sc"]
Oct 30 01:10:35.178992 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.178966    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 01:10:35.261262 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.261231    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-token\" (UniqueName: \"kubernetes.io/projected/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-token\") pod \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") "
Oct 30 01:10:35.261437 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.261291    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-envoy\") pod \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") "
Oct 30 01:10:35.261437 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.261322    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-workload-certs\") pod \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") "
Oct 30 01:10:35.261437 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.261360    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istiod-ca-cert\") pod \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") "
Oct 30 01:10:35.261437 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.261416    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b460b90d-0d55-44af-a4b9-2e70b3a7b679-kube-api-access-f4tmv\" (UniqueName: \"kubernetes.io/projected/b460b90d-0d55-44af-a4b9-2e70b3a7b679-kube-api-access-f4tmv\") pod \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") "
Oct 30 01:10:35.261690 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.261549    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-podinfo\") pod \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") "
Oct 30 01:10:35.261690 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.261582    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-workload-socket\") pod \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") "
Oct 30 01:10:35.261690 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.261672    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-credential-socket\") pod \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") "
Oct 30 01:10:35.261862 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.261725    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-data\") pod \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\" (UID: \"b460b90d-0d55-44af-a4b9-2e70b3a7b679\") "
Oct 30 01:10:35.261862 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.261724    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-workload-certs" pod "b460b90d-0d55-44af-a4b9-2e70b3a7b679" (UID: "b460b90d-0d55-44af-a4b9-2e70b3a7b679"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:35.261994 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.261972    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-workload-socket" pod "b460b90d-0d55-44af-a4b9-2e70b3a7b679" (UID: "b460b90d-0d55-44af-a4b9-2e70b3a7b679"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:35.262250 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.262219    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-data" pod "b460b90d-0d55-44af-a4b9-2e70b3a7b679" (UID: "b460b90d-0d55-44af-a4b9-2e70b3a7b679"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:35.262380 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.262365    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:35.262445 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.262382    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:35.262445 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.262376    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-credential-socket" pod "b460b90d-0d55-44af-a4b9-2e70b3a7b679" (UID: "b460b90d-0d55-44af-a4b9-2e70b3a7b679"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:35.262445 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.262392    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:35.262723 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.262699    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istiod-ca-cert" pod "b460b90d-0d55-44af-a4b9-2e70b3a7b679" (UID: "b460b90d-0d55-44af-a4b9-2e70b3a7b679"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:35.264170 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.264136    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-token" pod "b460b90d-0d55-44af-a4b9-2e70b3a7b679" (UID: "b460b90d-0d55-44af-a4b9-2e70b3a7b679"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:35.264663 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.264634    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-envoy" pod "b460b90d-0d55-44af-a4b9-2e70b3a7b679" (UID: "b460b90d-0d55-44af-a4b9-2e70b3a7b679"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:35.264978 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.264942    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-podinfo" pod "b460b90d-0d55-44af-a4b9-2e70b3a7b679" (UID: "b460b90d-0d55-44af-a4b9-2e70b3a7b679"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:10:35.265313 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.265289    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b460b90d-0d55-44af-a4b9-2e70b3a7b679-kube-api-access-f4tmv" pod "b460b90d-0d55-44af-a4b9-2e70b3a7b679" (UID: "b460b90d-0d55-44af-a4b9-2e70b3a7b679"). InnerVolumeSpecName "kube-api-access-f4tmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:35.363584 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.363505    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:35.363584 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.363532    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:35.363584 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.363541    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:35.363584 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.363550    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4tmv\" (UniqueName: \"kubernetes.io/projected/b460b90d-0d55-44af-a4b9-2e70b3a7b679-kube-api-access-f4tmv\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:35.363584 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.363561    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b460b90d-0d55-44af-a4b9-2e70b3a7b679-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:35.363584 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.363572    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b460b90d-0d55-44af-a4b9-2e70b3a7b679-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:35.759196 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.759157    4864 generic.go:356] "Generic (PLEG): container finished" podID="b460b90d-0d55-44af-a4b9-2e70b3a7b679" containerID="65b1a10ab81c4ae278a3dbb13c7a38f6ac885ce43be98ada7c28696af5455ad3" exitCode=0
Oct 30 01:10:35.759383 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.759225    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"
Oct 30 01:10:35.759383 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.759236    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d" event={"ID":"b460b90d-0d55-44af-a4b9-2e70b3a7b679","Type":"ContainerDied","Data":"65b1a10ab81c4ae278a3dbb13c7a38f6ac885ce43be98ada7c28696af5455ad3"}
Oct 30 01:10:35.759383 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.759300    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d" event={"ID":"b460b90d-0d55-44af-a4b9-2e70b3a7b679","Type":"ContainerDied","Data":"e54ac2f1eb406863c54f551cfa5649b6e39a83ec02a36d3ec5e459628d67bf62"}
Oct 30 01:10:35.759383 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.759330    4864 scope.go:122] "RemoveContainer" containerID="65b1a10ab81c4ae278a3dbb13c7a38f6ac885ce43be98ada7c28696af5455ad3"
Oct 30 01:10:35.770348 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.770326    4864 scope.go:122] "RemoveContainer" containerID="65b1a10ab81c4ae278a3dbb13c7a38f6ac885ce43be98ada7c28696af5455ad3"
Oct 30 01:10:35.770627 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:10:35.770604    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b1a10ab81c4ae278a3dbb13c7a38f6ac885ce43be98ada7c28696af5455ad3\": container with ID starting with 65b1a10ab81c4ae278a3dbb13c7a38f6ac885ce43be98ada7c28696af5455ad3 not found: ID does not exist" containerID="65b1a10ab81c4ae278a3dbb13c7a38f6ac885ce43be98ada7c28696af5455ad3"
Oct 30 01:10:35.770675 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.770637    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b1a10ab81c4ae278a3dbb13c7a38f6ac885ce43be98ada7c28696af5455ad3"} err="failed to get container status \"65b1a10ab81c4ae278a3dbb13c7a38f6ac885ce43be98ada7c28696af5455ad3\": rpc error: code = NotFound desc = could not find container \"65b1a10ab81c4ae278a3dbb13c7a38f6ac885ce43be98ada7c28696af5455ad3\": container with ID starting with 65b1a10ab81c4ae278a3dbb13c7a38f6ac885ce43be98ada7c28696af5455ad3 not found: ID does not exist"
Oct 30 01:10:35.792638 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.792611    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30401bdf-ea14-4729-9507-f81eb6685a43" path="/var/lib/kubelet/pods/30401bdf-ea14-4729-9507-f81eb6685a43/volumes"
Oct 30 01:10:35.806436 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.806412    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"]
Oct 30 01:10:35.816652 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:35.816632    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-mdq2d"]
Oct 30 01:10:37.725364 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:37.725331    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"]
Oct 30 01:10:37.725900 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:37.725552    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt" podUID="a8150dfe-249d-4cbe-9652-226dd18509f5" containerName="istio-proxy" containerID="cri-o://67dddb7f344b9aa5de80dc5c7a287ea70541a106bc9fee0090cd52cbbb985ef7" gracePeriod=30
Oct 30 01:10:37.793780 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:37.793748    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b460b90d-0d55-44af-a4b9-2e70b3a7b679" path="/var/lib/kubelet/pods/b460b90d-0d55-44af-a4b9-2e70b3a7b679/volumes"
Oct 30 01:10:42.988025 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:42.987988    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 01:10:43.168507 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.168414    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8150dfe-249d-4cbe-9652-226dd18509f5-kube-api-access-ntgtb\" (UniqueName: \"kubernetes.io/projected/a8150dfe-249d-4cbe-9652-226dd18509f5-kube-api-access-ntgtb\") pod \"a8150dfe-249d-4cbe-9652-226dd18509f5\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") "
Oct 30 01:10:43.168507 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.168493    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-workload-socket\") pod \"a8150dfe-249d-4cbe-9652-226dd18509f5\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") "
Oct 30 01:10:43.168724 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.168648    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8150dfe-249d-4cbe-9652-226dd18509f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8150dfe-249d-4cbe-9652-226dd18509f5-istiod-ca-cert\") pod \"a8150dfe-249d-4cbe-9652-226dd18509f5\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") "
Oct 30 01:10:43.168786 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.168727    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-workload-certs\") pod \"a8150dfe-249d-4cbe-9652-226dd18509f5\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") "
Oct 30 01:10:43.168786 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.168760    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-token\" (UniqueName: \"kubernetes.io/projected/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-token\") pod \"a8150dfe-249d-4cbe-9652-226dd18509f5\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") "
Oct 30 01:10:43.168892 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.168787    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-workload-socket" pod "a8150dfe-249d-4cbe-9652-226dd18509f5" (UID: "a8150dfe-249d-4cbe-9652-226dd18509f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:43.168892 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.168808    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-podinfo\") pod \"a8150dfe-249d-4cbe-9652-226dd18509f5\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") "
Oct 30 01:10:43.169000 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.168917    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-envoy\") pod \"a8150dfe-249d-4cbe-9652-226dd18509f5\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") "
Oct 30 01:10:43.169000 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.168972    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-data\") pod \"a8150dfe-249d-4cbe-9652-226dd18509f5\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") "
Oct 30 01:10:43.169156 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.168983    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-workload-certs" pod "a8150dfe-249d-4cbe-9652-226dd18509f5" (UID: "a8150dfe-249d-4cbe-9652-226dd18509f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:43.169156 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.169021    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8150dfe-249d-4cbe-9652-226dd18509f5-istiod-ca-cert" pod "a8150dfe-249d-4cbe-9652-226dd18509f5" (UID: "a8150dfe-249d-4cbe-9652-226dd18509f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:43.169156 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.169055    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-credential-socket\") pod \"a8150dfe-249d-4cbe-9652-226dd18509f5\" (UID: \"a8150dfe-249d-4cbe-9652-226dd18509f5\") "
Oct 30 01:10:43.169364 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.169340    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-data" pod "a8150dfe-249d-4cbe-9652-226dd18509f5" (UID: "a8150dfe-249d-4cbe-9652-226dd18509f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:43.169486 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.169467    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-credential-socket" pod "a8150dfe-249d-4cbe-9652-226dd18509f5" (UID: "a8150dfe-249d-4cbe-9652-226dd18509f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:43.169724 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.169706    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:43.169797 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.169726    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:43.169797 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.169736    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:43.169797 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.169745    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8150dfe-249d-4cbe-9652-226dd18509f5-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:43.169797 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.169758    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:43.170969 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.170941    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-token" pod "a8150dfe-249d-4cbe-9652-226dd18509f5" (UID: "a8150dfe-249d-4cbe-9652-226dd18509f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:43.170969 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.170960    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-podinfo" pod "a8150dfe-249d-4cbe-9652-226dd18509f5" (UID: "a8150dfe-249d-4cbe-9652-226dd18509f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:10:43.171229 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.171210    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8150dfe-249d-4cbe-9652-226dd18509f5-kube-api-access-ntgtb" pod "a8150dfe-249d-4cbe-9652-226dd18509f5" (UID: "a8150dfe-249d-4cbe-9652-226dd18509f5"). InnerVolumeSpecName "kube-api-access-ntgtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:43.171311 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.171290    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-envoy" pod "a8150dfe-249d-4cbe-9652-226dd18509f5" (UID: "a8150dfe-249d-4cbe-9652-226dd18509f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:43.270952 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.270922    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:43.270952 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.270949    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:43.270952 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.270958    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8150dfe-249d-4cbe-9652-226dd18509f5-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:43.271206 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.270966    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntgtb\" (UniqueName: \"kubernetes.io/projected/a8150dfe-249d-4cbe-9652-226dd18509f5-kube-api-access-ntgtb\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:10:43.824406 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.824377    4864 generic.go:356] "Generic (PLEG): container finished" podID="a8150dfe-249d-4cbe-9652-226dd18509f5" containerID="67dddb7f344b9aa5de80dc5c7a287ea70541a106bc9fee0090cd52cbbb985ef7" exitCode=0
Oct 30 01:10:43.824561 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.824423    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"
Oct 30 01:10:43.824561 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.824463    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt" event={"ID":"a8150dfe-249d-4cbe-9652-226dd18509f5","Type":"ContainerDied","Data":"67dddb7f344b9aa5de80dc5c7a287ea70541a106bc9fee0090cd52cbbb985ef7"}
Oct 30 01:10:43.824561 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.824518    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt" event={"ID":"a8150dfe-249d-4cbe-9652-226dd18509f5","Type":"ContainerDied","Data":"7e9574ffc042fd32bd92aa085d0239a6750c4e68467b8ca5110eb7c965115a98"}
Oct 30 01:10:43.824561 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.824539    4864 scope.go:122] "RemoveContainer" containerID="67dddb7f344b9aa5de80dc5c7a287ea70541a106bc9fee0090cd52cbbb985ef7"
Oct 30 01:10:43.834341 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.834321    4864 scope.go:122] "RemoveContainer" containerID="67dddb7f344b9aa5de80dc5c7a287ea70541a106bc9fee0090cd52cbbb985ef7"
Oct 30 01:10:43.834606 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:10:43.834587    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67dddb7f344b9aa5de80dc5c7a287ea70541a106bc9fee0090cd52cbbb985ef7\": container with ID starting with 67dddb7f344b9aa5de80dc5c7a287ea70541a106bc9fee0090cd52cbbb985ef7 not found: ID does not exist" containerID="67dddb7f344b9aa5de80dc5c7a287ea70541a106bc9fee0090cd52cbbb985ef7"
Oct 30 01:10:43.834674 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.834617    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67dddb7f344b9aa5de80dc5c7a287ea70541a106bc9fee0090cd52cbbb985ef7"} err="failed to get container status \"67dddb7f344b9aa5de80dc5c7a287ea70541a106bc9fee0090cd52cbbb985ef7\": rpc error: code = NotFound desc = could not find container \"67dddb7f344b9aa5de80dc5c7a287ea70541a106bc9fee0090cd52cbbb985ef7\": container with ID starting with 67dddb7f344b9aa5de80dc5c7a287ea70541a106bc9fee0090cd52cbbb985ef7 not found: ID does not exist"
Oct 30 01:10:43.872177 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.872139    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"]
Oct 30 01:10:43.881208 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.881184    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt"]
Oct 30 01:10:43.927113 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.920881    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:10:43.927113 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:43.920963    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-bjfvt" podUID="a8150dfe-249d-4cbe-9652-226dd18509f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:10:45.793358 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:10:45.793324    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8150dfe-249d-4cbe-9652-226dd18509f5" path="/var/lib/kubelet/pods/a8150dfe-249d-4cbe-9652-226dd18509f5/volumes"
Oct 30 01:10:49.040719 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:10:49.040683    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:11:13.149507 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:13.149469    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"]
Oct 30 01:11:13.149996 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:13.149709    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778" podUID="6dfc55de-0824-44de-a268-b2fcd2ea193f" containerName="istio-proxy" containerID="cri-o://7c2b2854b2d9fb031078ad57c308b5d530506a660f7ca74b503d2a944c659ae5" gracePeriod=30
Oct 30 01:11:13.159788 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:13.159754    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"]
Oct 30 01:11:13.160031 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:13.160005    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c" podUID="c08db946-5b81-477f-bc5a-8537b0385100" containerName="istio-proxy" containerID="cri-o://5a1f545e210433161c25127fa527a89e75af7995fc252f86da6e99a61ed3436d" gracePeriod=30
Oct 30 01:11:18.438411 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.438338    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 01:11:18.464694 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.464667    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 01:11:18.526465 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.526428    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-istio-data\") pod \"c08db946-5b81-477f-bc5a-8537b0385100\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") "
Oct 30 01:11:18.526670 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.526488    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-data\") pod \"6dfc55de-0824-44de-a268-b2fcd2ea193f\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") "
Oct 30 01:11:18.526670 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.526525    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dfc55de-0824-44de-a268-b2fcd2ea193f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dfc55de-0824-44de-a268-b2fcd2ea193f-istiod-ca-cert\") pod \"6dfc55de-0824-44de-a268-b2fcd2ea193f\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") "
Oct 30 01:11:18.526670 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.526585    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-credential-socket\") pod \"c08db946-5b81-477f-bc5a-8537b0385100\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") "
Oct 30 01:11:18.526670 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.526663    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-workload-certs\") pod \"6dfc55de-0824-44de-a268-b2fcd2ea193f\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") "
Oct 30 01:11:18.526897 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.526696    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c08db946-5b81-477f-bc5a-8537b0385100-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08db946-5b81-477f-bc5a-8537b0385100-istio-podinfo\") pod \"c08db946-5b81-477f-bc5a-8537b0385100\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") "
Oct 30 01:11:18.526897 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.526730    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-istio-envoy\") pod \"c08db946-5b81-477f-bc5a-8537b0385100\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") "
Oct 30 01:11:18.526897 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.526746    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-data" pod "6dfc55de-0824-44de-a268-b2fcd2ea193f" (UID: "6dfc55de-0824-44de-a268-b2fcd2ea193f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:18.526897 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.526766    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-credential-socket\") pod \"6dfc55de-0824-44de-a268-b2fcd2ea193f\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") "
Oct 30 01:11:18.526897 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.526745    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-istio-data" pod "c08db946-5b81-477f-bc5a-8537b0385100" (UID: "c08db946-5b81-477f-bc5a-8537b0385100"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:18.526897 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.526822    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c08db946-5b81-477f-bc5a-8537b0385100-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08db946-5b81-477f-bc5a-8537b0385100-istiod-ca-cert\") pod \"c08db946-5b81-477f-bc5a-8537b0385100\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") "
Oct 30 01:11:18.526897 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.526878    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-podinfo\") pod \"6dfc55de-0824-44de-a268-b2fcd2ea193f\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") "
Oct 30 01:11:18.527304 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.526936    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08db946-5b81-477f-bc5a-8537b0385100-kube-api-access-kcld5\" (UniqueName: \"kubernetes.io/projected/c08db946-5b81-477f-bc5a-8537b0385100-kube-api-access-kcld5\") pod \"c08db946-5b81-477f-bc5a-8537b0385100\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") "
Oct 30 01:11:18.527304 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.526967    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-workload-certs" pod "6dfc55de-0824-44de-a268-b2fcd2ea193f" (UID: "6dfc55de-0824-44de-a268-b2fcd2ea193f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:18.527304 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.526978    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-credential-socket" pod "c08db946-5b81-477f-bc5a-8537b0385100" (UID: "c08db946-5b81-477f-bc5a-8537b0385100"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:18.527304 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.526991    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-workload-certs\") pod \"c08db946-5b81-477f-bc5a-8537b0385100\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") "
Oct 30 01:11:18.527304 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.527025    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dfc55de-0824-44de-a268-b2fcd2ea193f-istiod-ca-cert" pod "6dfc55de-0824-44de-a268-b2fcd2ea193f" (UID: "6dfc55de-0824-44de-a268-b2fcd2ea193f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:18.527304 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.527052    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08db946-5b81-477f-bc5a-8537b0385100-istio-token\" (UniqueName: \"kubernetes.io/projected/c08db946-5b81-477f-bc5a-8537b0385100-istio-token\") pod \"c08db946-5b81-477f-bc5a-8537b0385100\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") "
Oct 30 01:11:18.527304 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.527162    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-workload-socket\") pod \"c08db946-5b81-477f-bc5a-8537b0385100\" (UID: \"c08db946-5b81-477f-bc5a-8537b0385100\") "
Oct 30 01:11:18.527304 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.527199    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-credential-socket" pod "6dfc55de-0824-44de-a268-b2fcd2ea193f" (UID: "6dfc55de-0824-44de-a268-b2fcd2ea193f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:18.527304 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.527233    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-workload-socket\") pod \"6dfc55de-0824-44de-a268-b2fcd2ea193f\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") "
Oct 30 01:11:18.527304 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.527279    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-token\" (UniqueName: \"kubernetes.io/projected/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-token\") pod \"6dfc55de-0824-44de-a268-b2fcd2ea193f\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") "
Oct 30 01:11:18.527804 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.527324    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-envoy\") pod \"6dfc55de-0824-44de-a268-b2fcd2ea193f\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") "
Oct 30 01:11:18.527804 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.527350    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dfc55de-0824-44de-a268-b2fcd2ea193f-kube-api-access-p897n\" (UniqueName: \"kubernetes.io/projected/6dfc55de-0824-44de-a268-b2fcd2ea193f-kube-api-access-p897n\") pod \"6dfc55de-0824-44de-a268-b2fcd2ea193f\" (UID: \"6dfc55de-0824-44de-a268-b2fcd2ea193f\") "
Oct 30 01:11:18.527804 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.527612    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08db946-5b81-477f-bc5a-8537b0385100-istiod-ca-cert" pod "c08db946-5b81-477f-bc5a-8537b0385100" (UID: "c08db946-5b81-477f-bc5a-8537b0385100"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:18.527804 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.527743    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-workload-socket" pod "c08db946-5b81-477f-bc5a-8537b0385100" (UID: "c08db946-5b81-477f-bc5a-8537b0385100"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:18.528007 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.527937    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-workload-socket" pod "6dfc55de-0824-44de-a268-b2fcd2ea193f" (UID: "6dfc55de-0824-44de-a268-b2fcd2ea193f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:18.528138 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.528117    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:11:18.528215 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.528143    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:11:18.528215 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.528158    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dfc55de-0824-44de-a268-b2fcd2ea193f-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:11:18.528215 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.528172    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:11:18.528215 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.528187    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:11:18.528215 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.528203    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:11:18.528479 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.528216    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08db946-5b81-477f-bc5a-8537b0385100-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:11:18.528479 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.528231    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:11:18.528479 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.528248    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:11:18.528479 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.528305    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-workload-certs" pod "c08db946-5b81-477f-bc5a-8537b0385100" (UID: "c08db946-5b81-477f-bc5a-8537b0385100"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:18.530481 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.530450    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-envoy" pod "6dfc55de-0824-44de-a268-b2fcd2ea193f" (UID: "6dfc55de-0824-44de-a268-b2fcd2ea193f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:18.530630 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.530567    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c08db946-5b81-477f-bc5a-8537b0385100-istio-podinfo" pod "c08db946-5b81-477f-bc5a-8537b0385100" (UID: "c08db946-5b81-477f-bc5a-8537b0385100"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:11:18.530891 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.530835    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-podinfo" pod "6dfc55de-0824-44de-a268-b2fcd2ea193f" (UID: "6dfc55de-0824-44de-a268-b2fcd2ea193f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:11:18.531004 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.530932    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08db946-5b81-477f-bc5a-8537b0385100-istio-token" pod "c08db946-5b81-477f-bc5a-8537b0385100" (UID: "c08db946-5b81-477f-bc5a-8537b0385100"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:18.531161 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.531144    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-token" pod "6dfc55de-0824-44de-a268-b2fcd2ea193f" (UID: "6dfc55de-0824-44de-a268-b2fcd2ea193f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:18.531904 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.531881    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08db946-5b81-477f-bc5a-8537b0385100-kube-api-access-kcld5" pod "c08db946-5b81-477f-bc5a-8537b0385100" (UID: "c08db946-5b81-477f-bc5a-8537b0385100"). InnerVolumeSpecName "kube-api-access-kcld5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:18.532044 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.532028    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-istio-envoy" pod "c08db946-5b81-477f-bc5a-8537b0385100" (UID: "c08db946-5b81-477f-bc5a-8537b0385100"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:18.532344 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.532325    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dfc55de-0824-44de-a268-b2fcd2ea193f-kube-api-access-p897n" pod "6dfc55de-0824-44de-a268-b2fcd2ea193f" (UID: "6dfc55de-0824-44de-a268-b2fcd2ea193f"). InnerVolumeSpecName "kube-api-access-p897n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:18.629707 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.629666    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:11:18.629707 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.629697    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:11:18.629707 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.629707    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p897n\" (UniqueName: \"kubernetes.io/projected/6dfc55de-0824-44de-a268-b2fcd2ea193f-kube-api-access-p897n\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:11:18.629707 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.629717    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08db946-5b81-477f-bc5a-8537b0385100-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:11:18.629980 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.629726    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:11:18.629980 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.629734    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dfc55de-0824-44de-a268-b2fcd2ea193f-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:11:18.629980 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.629743    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcld5\" (UniqueName: \"kubernetes.io/projected/c08db946-5b81-477f-bc5a-8537b0385100-kube-api-access-kcld5\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:11:18.629980 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.629753    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08db946-5b81-477f-bc5a-8537b0385100-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:11:18.629980 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:18.629763    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08db946-5b81-477f-bc5a-8537b0385100-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:11:19.040549 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:11:19.040512    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:11:19.075803 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:19.075766    4864 generic.go:356] "Generic (PLEG): container finished" podID="c08db946-5b81-477f-bc5a-8537b0385100" containerID="5a1f545e210433161c25127fa527a89e75af7995fc252f86da6e99a61ed3436d" exitCode=0
Oct 30 01:11:19.075972 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:19.075822    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"
Oct 30 01:11:19.075972 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:19.075842    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c" event={"ID":"c08db946-5b81-477f-bc5a-8537b0385100","Type":"ContainerDied","Data":"5a1f545e210433161c25127fa527a89e75af7995fc252f86da6e99a61ed3436d"}
Oct 30 01:11:19.075972 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:19.075900    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c" event={"ID":"c08db946-5b81-477f-bc5a-8537b0385100","Type":"ContainerDied","Data":"8a6d15f8d2f961c326d35650e88457d49a64b562584c02a635616cddbd96422d"}
Oct 30 01:11:19.075972 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:19.075931    4864 scope.go:122] "RemoveContainer" containerID="5a1f545e210433161c25127fa527a89e75af7995fc252f86da6e99a61ed3436d"
Oct 30 01:11:19.077427 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:19.077402    4864 generic.go:356] "Generic (PLEG): container finished" podID="6dfc55de-0824-44de-a268-b2fcd2ea193f" containerID="7c2b2854b2d9fb031078ad57c308b5d530506a660f7ca74b503d2a944c659ae5" exitCode=0
Oct 30 01:11:19.077552 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:19.077452    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"
Oct 30 01:11:19.077552 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:19.077502    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778" event={"ID":"6dfc55de-0824-44de-a268-b2fcd2ea193f","Type":"ContainerDied","Data":"7c2b2854b2d9fb031078ad57c308b5d530506a660f7ca74b503d2a944c659ae5"}
Oct 30 01:11:19.077679 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:19.077554    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778" event={"ID":"6dfc55de-0824-44de-a268-b2fcd2ea193f","Type":"ContainerDied","Data":"3266ba304dbfe49dda50485ce46213436709ac128cfb6caa233e4a465ff0c125"}
Oct 30 01:11:19.086835 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:19.086813    4864 scope.go:122] "RemoveContainer" containerID="026f8125a099382faf1e5739df3b1d06eb4ecec768b5049395b8d1f2edc8fd4a"
Oct 30 01:11:19.096281 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:19.096256    4864 scope.go:122] "RemoveContainer" containerID="5a1f545e210433161c25127fa527a89e75af7995fc252f86da6e99a61ed3436d"
Oct 30 01:11:19.096579 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:11:19.096562    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a1f545e210433161c25127fa527a89e75af7995fc252f86da6e99a61ed3436d\": container with ID starting with 5a1f545e210433161c25127fa527a89e75af7995fc252f86da6e99a61ed3436d not found: ID does not exist" containerID="5a1f545e210433161c25127fa527a89e75af7995fc252f86da6e99a61ed3436d"
Oct 30 01:11:19.096637 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:19.096587    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a1f545e210433161c25127fa527a89e75af7995fc252f86da6e99a61ed3436d"} err="failed to get container status \"5a1f545e210433161c25127fa527a89e75af7995fc252f86da6e99a61ed3436d\": rpc error: code = NotFound desc = could not find container \"5a1f545e210433161c25127fa527a89e75af7995fc252f86da6e99a61ed3436d\": container with ID starting with 5a1f545e210433161c25127fa527a89e75af7995fc252f86da6e99a61ed3436d not found: ID does not exist"
Oct 30 01:11:19.096637 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:19.096608    4864 scope.go:122] "RemoveContainer" containerID="026f8125a099382faf1e5739df3b1d06eb4ecec768b5049395b8d1f2edc8fd4a"
Oct 30 01:11:19.096858 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:11:19.096841    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"026f8125a099382faf1e5739df3b1d06eb4ecec768b5049395b8d1f2edc8fd4a\": container with ID starting with 026f8125a099382faf1e5739df3b1d06eb4ecec768b5049395b8d1f2edc8fd4a not found: ID does not exist" containerID="026f8125a099382faf1e5739df3b1d06eb4ecec768b5049395b8d1f2edc8fd4a"
Oct 30 01:11:19.096908 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:19.096866    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026f8125a099382faf1e5739df3b1d06eb4ecec768b5049395b8d1f2edc8fd4a"} err="failed to get container status \"026f8125a099382faf1e5739df3b1d06eb4ecec768b5049395b8d1f2edc8fd4a\": rpc error: code = NotFound desc = could not find container \"026f8125a099382faf1e5739df3b1d06eb4ecec768b5049395b8d1f2edc8fd4a\": container with ID starting with 026f8125a099382faf1e5739df3b1d06eb4ecec768b5049395b8d1f2edc8fd4a not found: ID does not exist"
Oct 30 01:11:19.096908 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:19.096882    4864 scope.go:122] "RemoveContainer" containerID="7c2b2854b2d9fb031078ad57c308b5d530506a660f7ca74b503d2a944c659ae5"
Oct 30 01:11:19.106373 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:19.106348    4864 scope.go:122] "RemoveContainer" containerID="7c2b2854b2d9fb031078ad57c308b5d530506a660f7ca74b503d2a944c659ae5"
Oct 30 01:11:19.106646 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:11:19.106620    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c2b2854b2d9fb031078ad57c308b5d530506a660f7ca74b503d2a944c659ae5\": container with ID starting with 7c2b2854b2d9fb031078ad57c308b5d530506a660f7ca74b503d2a944c659ae5 not found: ID does not exist" containerID="7c2b2854b2d9fb031078ad57c308b5d530506a660f7ca74b503d2a944c659ae5"
Oct 30 01:11:19.106725 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:19.106655    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2b2854b2d9fb031078ad57c308b5d530506a660f7ca74b503d2a944c659ae5"} err="failed to get container status \"7c2b2854b2d9fb031078ad57c308b5d530506a660f7ca74b503d2a944c659ae5\": rpc error: code = NotFound desc = could not find container \"7c2b2854b2d9fb031078ad57c308b5d530506a660f7ca74b503d2a944c659ae5\": container with ID starting with 7c2b2854b2d9fb031078ad57c308b5d530506a660f7ca74b503d2a944c659ae5 not found: ID does not exist"
Oct 30 01:11:19.124952 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:19.124922    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"]
Oct 30 01:11:19.134838 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:19.134809    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-pqh6c"]
Oct 30 01:11:19.164182 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:19.164133    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"]
Oct 30 01:11:19.166805 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:19.166778    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-f8778"]
Oct 30 01:11:19.794293 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:19.794260    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dfc55de-0824-44de-a268-b2fcd2ea193f" path="/var/lib/kubelet/pods/6dfc55de-0824-44de-a268-b2fcd2ea193f/volumes"
Oct 30 01:11:19.794766 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:19.794751    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c08db946-5b81-477f-bc5a-8537b0385100" path="/var/lib/kubelet/pods/c08db946-5b81-477f-bc5a-8537b0385100/volumes"
Oct 30 01:11:23.264932 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.264900    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"]
Oct 30 01:11:23.265650 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.265629    4864 cpu_manager.go:414] "containerMap: removing container" podUID="c136a701-915b-409a-8b28-198deed55a38" containerName="istio-proxy"
Oct 30 01:11:23.265772 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.265653    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c136a701-915b-409a-8b28-198deed55a38" containerName="istio-proxy"
Oct 30 01:11:23.265772 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.265673    4864 cpu_manager.go:414] "containerMap: removing container" podUID="c08db946-5b81-477f-bc5a-8537b0385100" containerName="istio-proxy"
Oct 30 01:11:23.265772 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.265680    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08db946-5b81-477f-bc5a-8537b0385100" containerName="istio-proxy"
Oct 30 01:11:23.265772 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.265698    4864 cpu_manager.go:414] "containerMap: removing container" podUID="6dfc55de-0824-44de-a268-b2fcd2ea193f" containerName="istio-proxy"
Oct 30 01:11:23.265772 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.265705    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dfc55de-0824-44de-a268-b2fcd2ea193f" containerName="istio-proxy"
Oct 30 01:11:23.265772 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.265735    4864 cpu_manager.go:414] "containerMap: removing container" podUID="c08db946-5b81-477f-bc5a-8537b0385100" containerName="istio-proxy"
Oct 30 01:11:23.265772 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.265742    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08db946-5b81-477f-bc5a-8537b0385100" containerName="istio-proxy"
Oct 30 01:11:23.265772 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.265755    4864 cpu_manager.go:414] "containerMap: removing container" podUID="b460b90d-0d55-44af-a4b9-2e70b3a7b679" containerName="istio-proxy"
Oct 30 01:11:23.265772 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.265762    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b460b90d-0d55-44af-a4b9-2e70b3a7b679" containerName="istio-proxy"
Oct 30 01:11:23.265772 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.265774    4864 cpu_manager.go:414] "containerMap: removing container" podUID="30401bdf-ea14-4729-9507-f81eb6685a43" containerName="istio-proxy"
Oct 30 01:11:23.266330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.265781    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30401bdf-ea14-4729-9507-f81eb6685a43" containerName="istio-proxy"
Oct 30 01:11:23.266330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.265790    4864 cpu_manager.go:414] "containerMap: removing container" podUID="30401bdf-ea14-4729-9507-f81eb6685a43" containerName="istio-proxy"
Oct 30 01:11:23.266330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.265797    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30401bdf-ea14-4729-9507-f81eb6685a43" containerName="istio-proxy"
Oct 30 01:11:23.266330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.265810    4864 cpu_manager.go:414] "containerMap: removing container" podUID="67f83fb2-248a-49c8-ba87-ee314ff0006e" containerName="istio-proxy"
Oct 30 01:11:23.266330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.265817    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67f83fb2-248a-49c8-ba87-ee314ff0006e" containerName="istio-proxy"
Oct 30 01:11:23.266330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.265837    4864 cpu_manager.go:414] "containerMap: removing container" podUID="370c4c48-476a-45b1-bb4b-09f1d92d0fa1" containerName="istio-proxy"
Oct 30 01:11:23.266330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.265844    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="370c4c48-476a-45b1-bb4b-09f1d92d0fa1" containerName="istio-proxy"
Oct 30 01:11:23.266330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.265867    4864 cpu_manager.go:414] "containerMap: removing container" podUID="a8150dfe-249d-4cbe-9652-226dd18509f5" containerName="istio-proxy"
Oct 30 01:11:23.266330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.265874    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8150dfe-249d-4cbe-9652-226dd18509f5" containerName="istio-proxy"
Oct 30 01:11:23.266330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.265900    4864 cpu_manager.go:414] "containerMap: removing container" podUID="7c965c38-598d-47ce-9104-5a0b85c6a925" containerName="istio-proxy"
Oct 30 01:11:23.266330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.265908    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c965c38-598d-47ce-9104-5a0b85c6a925" containerName="istio-proxy"
Oct 30 01:11:23.266330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.266044    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08db946-5b81-477f-bc5a-8537b0385100" containerName="istio-proxy"
Oct 30 01:11:23.266330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.266058    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="b460b90d-0d55-44af-a4b9-2e70b3a7b679" containerName="istio-proxy"
Oct 30 01:11:23.266330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.266087    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="30401bdf-ea14-4729-9507-f81eb6685a43" containerName="istio-proxy"
Oct 30 01:11:23.266330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.266097    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="30401bdf-ea14-4729-9507-f81eb6685a43" containerName="istio-proxy"
Oct 30 01:11:23.266330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.266107    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="67f83fb2-248a-49c8-ba87-ee314ff0006e" containerName="istio-proxy"
Oct 30 01:11:23.266330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.266118    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="370c4c48-476a-45b1-bb4b-09f1d92d0fa1" containerName="istio-proxy"
Oct 30 01:11:23.266330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.266128    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8150dfe-249d-4cbe-9652-226dd18509f5" containerName="istio-proxy"
Oct 30 01:11:23.266330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.266141    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c965c38-598d-47ce-9104-5a0b85c6a925" containerName="istio-proxy"
Oct 30 01:11:23.266330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.266152    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="c136a701-915b-409a-8b28-198deed55a38" containerName="istio-proxy"
Oct 30 01:11:23.266330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.266162    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08db946-5b81-477f-bc5a-8537b0385100" containerName="istio-proxy"
Oct 30 01:11:23.266330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.266172    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dfc55de-0824-44de-a268-b2fcd2ea193f" containerName="istio-proxy"
Oct 30 01:11:23.269861 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.269839    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.283276 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.283251    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-1d6d47-openshift-default-dockercfg-6rb67\""
Oct 30 01:11:23.297850 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.297816    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"]
Oct 30 01:11:23.394976 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.394937    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-istio-envoy\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.395180 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.394991    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ea3935-4494-42cc-91e4-71f269edaee6-istio-token\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.395180 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.395083    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-workload-certs\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.395307 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.395179    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ea3935-4494-42cc-91e4-71f269edaee6-istio-podinfo\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.395307 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.395285    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-istio-data\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.395416 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.395322    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ea3935-4494-42cc-91e4-71f269edaee6-istiod-ca-cert\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.395474 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.395417    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fffsm\" (UniqueName: \"kubernetes.io/projected/29ea3935-4494-42cc-91e4-71f269edaee6-kube-api-access-fffsm\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.395587 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.395572    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-workload-socket\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.395649 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.395612    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-credential-socket\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.497216 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.497178    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ea3935-4494-42cc-91e4-71f269edaee6-istio-token\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.497518 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.497221    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-workload-certs\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.497518 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.497260    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ea3935-4494-42cc-91e4-71f269edaee6-istio-podinfo\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.497518 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.497323    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-istio-data\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.497518 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.497437    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ea3935-4494-42cc-91e4-71f269edaee6-istiod-ca-cert\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.497864 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.497839    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-istio-data\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.498157 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.498133    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-workload-certs\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.498263 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.498177    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ea3935-4494-42cc-91e4-71f269edaee6-istiod-ca-cert\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.498263 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.498241    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fffsm\" (UniqueName: \"kubernetes.io/projected/29ea3935-4494-42cc-91e4-71f269edaee6-kube-api-access-fffsm\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.498425 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.498407    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-workload-socket\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.498502 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.498457    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-credential-socket\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.498560 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.498512    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-istio-envoy\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.498760 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.498733    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-workload-socket\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.498988 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.498970    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-credential-socket\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.500775 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.500751    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ea3935-4494-42cc-91e4-71f269edaee6-istio-podinfo\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.501015 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.500996    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-istio-envoy\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.523862 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.523584    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ea3935-4494-42cc-91e4-71f269edaee6-istio-token\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.528259 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.528145    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fffsm\" (UniqueName: \"kubernetes.io/projected/29ea3935-4494-42cc-91e4-71f269edaee6-kube-api-access-fffsm\") pod \"e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") " pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.584693 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.584661    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:23.904975 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.904944    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"]
Oct 30 01:11:23.906247 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:11:23.906215    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ea3935_4494_42cc_91e4_71f269edaee6.slice/crio-c060a571389c47574be5050278b8d178af8df1b8ad03b9436a66cb8bc5b3fdd9 WatchSource:0}: Error finding container c060a571389c47574be5050278b8d178af8df1b8ad03b9436a66cb8bc5b3fdd9: Status 404 returned error can't find the container with id c060a571389c47574be5050278b8d178af8df1b8ad03b9436a66cb8bc5b3fdd9
Oct 30 01:11:23.908934 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.908896    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:11:23.909055 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.908971    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:11:23.909055 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:23.909005    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:11:24.113457 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:24.113424    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv" event={"ID":"29ea3935-4494-42cc-91e4-71f269edaee6","Type":"ContainerStarted","Data":"3efe45031f427a495fe0ee48ae19de611e42e23e7c0571108cce71e0b7667df0"}
Oct 30 01:11:24.113625 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:24.113481    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv" event={"ID":"29ea3935-4494-42cc-91e4-71f269edaee6","Type":"ContainerStarted","Data":"c060a571389c47574be5050278b8d178af8df1b8ad03b9436a66cb8bc5b3fdd9"}
Oct 30 01:11:24.156393 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:24.156287    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv" podStartSLOduration=1.156272217 podStartE2EDuration="1.156272217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:24.154460161 +0000 UTC m=+1796.492562071" watchObservedRunningTime="2026-10-30 01:11:24.156272217 +0000 UTC m=+1796.494374128"
Oct 30 01:11:24.585146 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:24.585107    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:25.591178 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:25.591148    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:26.125557 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:26.125530    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:26.126443 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:26.126424    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:11:28.037661 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:28.037633    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:11:28.045312 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:28.045283    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:11:49.054841 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:11:49.054794    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:11:49.484979 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:49.484945    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl"]
Oct 30 01:11:49.488979 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:49.488954    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl"
Oct 30 01:11:49.506392 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:49.506339    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-1d6d47-serving-cert\""
Oct 30 01:11:49.518405 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:49.518373    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl"]
Oct 30 01:11:49.630757 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:49.630717    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gzdp\" (UniqueName: \"kubernetes.io/projected/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-kube-api-access-8gzdp\") pod \"maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl\" (UID: \"3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl"
Oct 30 01:11:49.630938 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:49.630783    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-metrics-certs\") pod \"maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl\" (UID: \"3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl"
Oct 30 01:11:49.630938 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:49.630902    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl\" (UID: \"3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl"
Oct 30 01:11:49.631136 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:49.631120    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl\" (UID: \"3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl"
Oct 30 01:11:49.737145 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:49.733010    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gzdp\" (UniqueName: \"kubernetes.io/projected/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-kube-api-access-8gzdp\") pod \"maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl\" (UID: \"3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl"
Oct 30 01:11:49.737145 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:49.733106    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-metrics-certs\") pod \"maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl\" (UID: \"3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl"
Oct 30 01:11:49.737145 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:49.733181    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl\" (UID: \"3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl"
Oct 30 01:11:49.737145 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:49.733319    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl\" (UID: \"3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl"
Oct 30 01:11:49.737145 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:49.734378    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl\" (UID: \"3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl"
Oct 30 01:11:49.747105 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:49.746624    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-metrics-certs\") pod \"maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl\" (UID: \"3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl"
Oct 30 01:11:49.772093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:49.762387    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl\" (UID: \"3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl"
Oct 30 01:11:49.775181 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:49.775147    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gzdp\" (UniqueName: \"kubernetes.io/projected/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-kube-api-access-8gzdp\") pod \"maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl\" (UID: \"3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl"
Oct 30 01:11:49.806446 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:49.806411    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl"
Oct 30 01:11:50.128346 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:50.128262    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl"]
Oct 30 01:11:50.139147 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:11:50.139112    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf60d4d_16e5_46c7_9dc4_d6ecad8b9816.slice/crio-a5d3a8f7003b99c22eb9397be4ea81d7a89a0b5fd42585dc9191a21e43cf7bfb WatchSource:0}: Error finding container a5d3a8f7003b99c22eb9397be4ea81d7a89a0b5fd42585dc9191a21e43cf7bfb: Status 404 returned error can't find the container with id a5d3a8f7003b99c22eb9397be4ea81d7a89a0b5fd42585dc9191a21e43cf7bfb
Oct 30 01:11:50.276285 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:50.276243    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl" event={"ID":"3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816","Type":"ContainerStarted","Data":"a5d3a8f7003b99c22eb9397be4ea81d7a89a0b5fd42585dc9191a21e43cf7bfb"}
Oct 30 01:11:51.287045 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:51.287003    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl" event={"ID":"3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816","Type":"ContainerStarted","Data":"9a4a97427e06f49beab511cbff1b7ba7960fb2e13eef57686ae560334aee853c"}
Oct 30 01:11:51.287456 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:51.287238    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl"
Oct 30 01:11:51.330263 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:51.328667    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl" podStartSLOduration=1.928343524 podStartE2EDuration="2.328647816s" totalImagesPullingTime="400.304292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:51.325065016 +0000 UTC m=+1823.663166927" watchObservedRunningTime="2026-10-30 01:11:51.328647816 +0000 UTC m=+1823.666749728"
Oct 30 01:11:54.256488 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.256452    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"]
Oct 30 01:11:54.260581 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.260563    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.285357 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.285329    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"]
Oct 30 01:11:54.399039 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.399004    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.399226 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.399066    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.399226 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.399206    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.399349 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.399248    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.399349 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.399275    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.399349 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.399334    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.399462 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.399381    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb21179a-656e-4c81-8c3e-fa670ec27c22-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.399462 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.399450    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q22vs\" (UniqueName: \"kubernetes.io/projected/fb21179a-656e-4c81-8c3e-fa670ec27c22-kube-api-access-q22vs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.399536 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.399524    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.500637 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.500595    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.500834 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.500743    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.500834 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.500793    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.500941 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.500857    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.500941 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.500898    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.501051 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.500936    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.501051 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.500960    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.501051 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.500987    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb21179a-656e-4c81-8c3e-fa670ec27c22-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.501051 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.501014    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q22vs\" (UniqueName: \"kubernetes.io/projected/fb21179a-656e-4c81-8c3e-fa670ec27c22-kube-api-access-q22vs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.501935 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.501278    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.501935 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.501313    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.501935 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.501354    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.501935 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.501388    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.502275 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.501959    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb21179a-656e-4c81-8c3e-fa670ec27c22-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.503348 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.503317    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.503590 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.503569    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.523245 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.523189    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.523347 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.523321    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q22vs\" (UniqueName: \"kubernetes.io/projected/fb21179a-656e-4c81-8c3e-fa670ec27c22-kube-api-access-q22vs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.577519 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.577477    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:54.893563 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.893538    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"]
Oct 30 01:11:54.895750 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:11:54.895722    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb21179a_656e_4c81_8c3e_fa670ec27c22.slice/crio-e27d6a1924406dc07dd603e00834263f2f56df645b2a450f440abd4fed64fc4d WatchSource:0}: Error finding container e27d6a1924406dc07dd603e00834263f2f56df645b2a450f440abd4fed64fc4d: Status 404 returned error can't find the container with id e27d6a1924406dc07dd603e00834263f2f56df645b2a450f440abd4fed64fc4d
Oct 30 01:11:54.898954 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.898917    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:11:54.899085 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.898985    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:11:54.899085 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:54.899012    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:11:55.313311 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:55.313277    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p" event={"ID":"fb21179a-656e-4c81-8c3e-fa670ec27c22","Type":"ContainerStarted","Data":"ce3532a6c368b5327eefd8eb672b4aec94b409a46cfb0d36b1c1836dd42d4805"}
Oct 30 01:11:55.313704 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:55.313328    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p" event={"ID":"fb21179a-656e-4c81-8c3e-fa670ec27c22","Type":"ContainerStarted","Data":"e27d6a1924406dc07dd603e00834263f2f56df645b2a450f440abd4fed64fc4d"}
Oct 30 01:11:55.357008 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:55.356952    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p" podStartSLOduration=1.356935107 podStartE2EDuration="1.356935107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:55.354219201 +0000 UTC m=+1827.692321111" watchObservedRunningTime="2026-10-30 01:11:55.356935107 +0000 UTC m=+1827.695037013"
Oct 30 01:11:55.578380 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:55.578306    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:11:55.580427 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:55.580393    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 01:11:55.580570 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:55.580451    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p" podUID="fb21179a-656e-4c81-8c3e-fa670ec27c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 01:11:56.581611 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:56.581559    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 01:11:56.582036 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:56.581641    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p" podUID="fb21179a-656e-4c81-8c3e-fa670ec27c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 01:11:56.634715 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:56.634675    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"]
Oct 30 01:11:56.634949 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:56.634922    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx" podUID="79304a2e-fd29-4e09-acd9-4d727ae11a48" containerName="istio-proxy" containerID="cri-o://2466e28668c8ed5ac454badcc15187aa24bdadb5df29732f321e5ad14f38c7d3" gracePeriod=30
Oct 30 01:11:56.687772 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:56.687739    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6k58k_d2a355c3-0703-422f-8ddb-555c4a364ade/payload-processing/0.log"
Oct 30 01:11:56.901187 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:56.901093    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6k58k_d2a355c3-0703-422f-8ddb-555c4a364ade/payload-processing/0.log"
Oct 30 01:11:57.299722 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:57.299689    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl"
Oct 30 01:11:57.578983 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:57.578886    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 01:11:57.579277 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:57.579249    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p" podUID="fb21179a-656e-4c81-8c3e-fa670ec27c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 01:11:58.580095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:58.579498    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 01:11:58.580095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:58.579581    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p" podUID="fb21179a-656e-4c81-8c3e-fa670ec27c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 01:11:59.580501 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:59.580461    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 01:11:59.581121 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:59.581064    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p" podUID="fb21179a-656e-4c81-8c3e-fa670ec27c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 01:11:59.644786 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:59.644736    4864 patch_prober.go:29] interesting pod/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:11:59.644991 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:11:59.644832    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx" podUID="79304a2e-fd29-4e09-acd9-4d727ae11a48" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:12:00.579486 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:00.579451    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 01:12:00.580516 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:00.580484    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p" podUID="fb21179a-656e-4c81-8c3e-fa670ec27c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 01:12:01.578813 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:01.578776    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 01:12:01.579135 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:01.579066    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p" podUID="fb21179a-656e-4c81-8c3e-fa670ec27c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 01:12:01.908311 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:01.908282    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:12:02.019780 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.019751    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-workload-certs\") pod \"79304a2e-fd29-4e09-acd9-4d727ae11a48\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") "
Oct 30 01:12:02.019780 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.019785    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79304a2e-fd29-4e09-acd9-4d727ae11a48-kube-api-access-w92wp\" (UniqueName: \"kubernetes.io/projected/79304a2e-fd29-4e09-acd9-4d727ae11a48-kube-api-access-w92wp\") pod \"79304a2e-fd29-4e09-acd9-4d727ae11a48\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") "
Oct 30 01:12:02.020018 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.019815    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-podinfo\") pod \"79304a2e-fd29-4e09-acd9-4d727ae11a48\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") "
Oct 30 01:12:02.020018 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.019990    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79304a2e-fd29-4e09-acd9-4d727ae11a48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79304a2e-fd29-4e09-acd9-4d727ae11a48-istiod-ca-cert\") pod \"79304a2e-fd29-4e09-acd9-4d727ae11a48\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") "
Oct 30 01:12:02.020018 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.020015    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-envoy\") pod \"79304a2e-fd29-4e09-acd9-4d727ae11a48\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") "
Oct 30 01:12:02.020239 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.020064    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-credential-socket\") pod \"79304a2e-fd29-4e09-acd9-4d727ae11a48\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") "
Oct 30 01:12:02.020239 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.020129    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-workload-certs" pod "79304a2e-fd29-4e09-acd9-4d727ae11a48" (UID: "79304a2e-fd29-4e09-acd9-4d727ae11a48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:02.020376 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.020351    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-credential-socket" pod "79304a2e-fd29-4e09-acd9-4d727ae11a48" (UID: "79304a2e-fd29-4e09-acd9-4d727ae11a48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:02.020444 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.020425    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-workload-socket\") pod \"79304a2e-fd29-4e09-acd9-4d727ae11a48\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") "
Oct 30 01:12:02.020554 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.020535    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-data\") pod \"79304a2e-fd29-4e09-acd9-4d727ae11a48\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") "
Oct 30 01:12:02.020658 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.020637    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79304a2e-fd29-4e09-acd9-4d727ae11a48-istiod-ca-cert" pod "79304a2e-fd29-4e09-acd9-4d727ae11a48" (UID: "79304a2e-fd29-4e09-acd9-4d727ae11a48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:02.020723 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.020638    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-workload-socket" pod "79304a2e-fd29-4e09-acd9-4d727ae11a48" (UID: "79304a2e-fd29-4e09-acd9-4d727ae11a48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:02.020801 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.020784    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-data" pod "79304a2e-fd29-4e09-acd9-4d727ae11a48" (UID: "79304a2e-fd29-4e09-acd9-4d727ae11a48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:02.020918 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.020894    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-token\" (UniqueName: \"kubernetes.io/projected/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-token\") pod \"79304a2e-fd29-4e09-acd9-4d727ae11a48\" (UID: \"79304a2e-fd29-4e09-acd9-4d727ae11a48\") "
Oct 30 01:12:02.021572 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.021545    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:12:02.021750 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.021735    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:12:02.021855 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.021841    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:12:02.021948 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.021936    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79304a2e-fd29-4e09-acd9-4d727ae11a48-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:12:02.022036 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.022024    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:12:02.023266 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.023232    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79304a2e-fd29-4e09-acd9-4d727ae11a48-kube-api-access-w92wp" pod "79304a2e-fd29-4e09-acd9-4d727ae11a48" (UID: "79304a2e-fd29-4e09-acd9-4d727ae11a48"). InnerVolumeSpecName "kube-api-access-w92wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:02.023266 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.023252    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-envoy" pod "79304a2e-fd29-4e09-acd9-4d727ae11a48" (UID: "79304a2e-fd29-4e09-acd9-4d727ae11a48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:02.023611 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.023577    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-podinfo" pod "79304a2e-fd29-4e09-acd9-4d727ae11a48" (UID: "79304a2e-fd29-4e09-acd9-4d727ae11a48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:12:02.023715 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.023699    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-token" pod "79304a2e-fd29-4e09-acd9-4d727ae11a48" (UID: "79304a2e-fd29-4e09-acd9-4d727ae11a48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:02.123814 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.123731    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:12:02.123814 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.123763    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:12:02.123814 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.123773    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w92wp\" (UniqueName: \"kubernetes.io/projected/79304a2e-fd29-4e09-acd9-4d727ae11a48-kube-api-access-w92wp\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:12:02.123814 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.123781    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79304a2e-fd29-4e09-acd9-4d727ae11a48-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:12:02.359780 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.359748    4864 generic.go:356] "Generic (PLEG): container finished" podID="79304a2e-fd29-4e09-acd9-4d727ae11a48" containerID="2466e28668c8ed5ac454badcc15187aa24bdadb5df29732f321e5ad14f38c7d3" exitCode=0
Oct 30 01:12:02.359942 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.359792    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx" event={"ID":"79304a2e-fd29-4e09-acd9-4d727ae11a48","Type":"ContainerDied","Data":"2466e28668c8ed5ac454badcc15187aa24bdadb5df29732f321e5ad14f38c7d3"}
Oct 30 01:12:02.359942 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.359817    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"
Oct 30 01:12:02.359942 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.359835    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx" event={"ID":"79304a2e-fd29-4e09-acd9-4d727ae11a48","Type":"ContainerDied","Data":"315d3398e18c22a9017d677a7d0072a9d6ba6e2a5415b32c44e2229fd4a94323"}
Oct 30 01:12:02.359942 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.359856    4864 scope.go:122] "RemoveContainer" containerID="2466e28668c8ed5ac454badcc15187aa24bdadb5df29732f321e5ad14f38c7d3"
Oct 30 01:12:02.370613 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.370590    4864 scope.go:122] "RemoveContainer" containerID="51c022560da22957b55edcc20c0f4f7d9bff806cce51ed570c805f486e1ec998"
Oct 30 01:12:02.380335 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.380311    4864 scope.go:122] "RemoveContainer" containerID="2466e28668c8ed5ac454badcc15187aa24bdadb5df29732f321e5ad14f38c7d3"
Oct 30 01:12:02.380647 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:12:02.380625    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2466e28668c8ed5ac454badcc15187aa24bdadb5df29732f321e5ad14f38c7d3\": container with ID starting with 2466e28668c8ed5ac454badcc15187aa24bdadb5df29732f321e5ad14f38c7d3 not found: ID does not exist" containerID="2466e28668c8ed5ac454badcc15187aa24bdadb5df29732f321e5ad14f38c7d3"
Oct 30 01:12:02.380695 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.380659    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2466e28668c8ed5ac454badcc15187aa24bdadb5df29732f321e5ad14f38c7d3"} err="failed to get container status \"2466e28668c8ed5ac454badcc15187aa24bdadb5df29732f321e5ad14f38c7d3\": rpc error: code = NotFound desc = could not find container \"2466e28668c8ed5ac454badcc15187aa24bdadb5df29732f321e5ad14f38c7d3\": container with ID starting with 2466e28668c8ed5ac454badcc15187aa24bdadb5df29732f321e5ad14f38c7d3 not found: ID does not exist"
Oct 30 01:12:02.380695 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.380677    4864 scope.go:122] "RemoveContainer" containerID="51c022560da22957b55edcc20c0f4f7d9bff806cce51ed570c805f486e1ec998"
Oct 30 01:12:02.380961 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:12:02.380936    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51c022560da22957b55edcc20c0f4f7d9bff806cce51ed570c805f486e1ec998\": container with ID starting with 51c022560da22957b55edcc20c0f4f7d9bff806cce51ed570c805f486e1ec998 not found: ID does not exist" containerID="51c022560da22957b55edcc20c0f4f7d9bff806cce51ed570c805f486e1ec998"
Oct 30 01:12:02.381021 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.380974    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51c022560da22957b55edcc20c0f4f7d9bff806cce51ed570c805f486e1ec998"} err="failed to get container status \"51c022560da22957b55edcc20c0f4f7d9bff806cce51ed570c805f486e1ec998\": rpc error: code = NotFound desc = could not find container \"51c022560da22957b55edcc20c0f4f7d9bff806cce51ed570c805f486e1ec998\": container with ID starting with 51c022560da22957b55edcc20c0f4f7d9bff806cce51ed570c805f486e1ec998 not found: ID does not exist"
Oct 30 01:12:02.408560 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.408533    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"]
Oct 30 01:12:02.420708 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.420681    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-4ec143-openshift-default-6b8f97b49-tkczx"]
Oct 30 01:12:02.599065 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.599038    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:12:02.599599 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.599553    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:12:02.600566 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:02.600548    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:12:03.793998 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:03.793960    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79304a2e-fd29-4e09-acd9-4d727ae11a48" path="/var/lib/kubelet/pods/79304a2e-fd29-4e09-acd9-4d727ae11a48/volumes"
Oct 30 01:12:19.040124 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:12:19.040028    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:12:24.519822 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:24.519785    4864 generic.go:356] "Generic (PLEG): container finished" podID="29ea3935-4494-42cc-91e4-71f269edaee6" containerID="3efe45031f427a495fe0ee48ae19de611e42e23e7c0571108cce71e0b7667df0" exitCode=0
Oct 30 01:12:24.520339 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:24.519860    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv" event={"ID":"29ea3935-4494-42cc-91e4-71f269edaee6","Type":"ContainerDied","Data":"3efe45031f427a495fe0ee48ae19de611e42e23e7c0571108cce71e0b7667df0"}
Oct 30 01:12:24.520386 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:24.520338    4864 scope.go:122] "RemoveContainer" containerID="3efe45031f427a495fe0ee48ae19de611e42e23e7c0571108cce71e0b7667df0"
Oct 30 01:12:24.521901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:24.521875    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:12:24.521960 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:24.521951    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:12:24.521998 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:24.521987    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:12:25.527844 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:25.527810    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv" event={"ID":"29ea3935-4494-42cc-91e4-71f269edaee6","Type":"ContainerStarted","Data":"8e066bec544a9a6194cd159bbec2c877af35e0dd4191685e712d4f24e6cf69f9"}
Oct 30 01:12:25.589093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:25.588165    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:12:25.591103 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:25.591047    4864 patch_prober.go:29] interesting pod/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 01:12:25.591219 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:25.591139    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv" podUID="29ea3935-4494-42cc-91e4-71f269edaee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 01:12:26.126157 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:26.126126    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:12:26.585673 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:26.585630    4864 patch_prober.go:29] interesting pod/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 01:12:26.586214 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:26.585711    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv" podUID="29ea3935-4494-42cc-91e4-71f269edaee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 01:12:27.585653 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:27.585608    4864 patch_prober.go:29] interesting pod/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 01:12:27.585944 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:27.585680    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv" podUID="29ea3935-4494-42cc-91e4-71f269edaee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 01:12:28.587579 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:28.587477    4864 patch_prober.go:29] interesting pod/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 01:12:28.587579 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:28.587554    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv" podUID="29ea3935-4494-42cc-91e4-71f269edaee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 01:12:29.587488 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:29.587443    4864 patch_prober.go:29] interesting pod/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 01:12:29.587821 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:29.587787    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv" podUID="29ea3935-4494-42cc-91e4-71f269edaee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 01:12:30.585106 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:30.585014    4864 patch_prober.go:29] interesting pod/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 01:12:30.585106 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:30.585082    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv" podUID="29ea3935-4494-42cc-91e4-71f269edaee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 01:12:31.590161 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:31.590130    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:12:31.591219 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:31.591202    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:12:39.398683 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.398644    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"]
Oct 30 01:12:39.399442 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.399421    4864 cpu_manager.go:414] "containerMap: removing container" podUID="79304a2e-fd29-4e09-acd9-4d727ae11a48" containerName="istio-proxy"
Oct 30 01:12:39.399559 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.399444    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79304a2e-fd29-4e09-acd9-4d727ae11a48" containerName="istio-proxy"
Oct 30 01:12:39.399559 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.399464    4864 cpu_manager.go:414] "containerMap: removing container" podUID="79304a2e-fd29-4e09-acd9-4d727ae11a48" containerName="istio-proxy"
Oct 30 01:12:39.399559 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.399471    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79304a2e-fd29-4e09-acd9-4d727ae11a48" containerName="istio-proxy"
Oct 30 01:12:39.399715 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.399663    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="79304a2e-fd29-4e09-acd9-4d727ae11a48" containerName="istio-proxy"
Oct 30 01:12:39.399715 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.399680    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="79304a2e-fd29-4e09-acd9-4d727ae11a48" containerName="istio-proxy"
Oct 30 01:12:39.402967 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.402946    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.434762 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.434734    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"]
Oct 30 01:12:39.503398 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.503360    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.503583 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.503432    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc29530c-080c-4698-a2f3-7300a77e9804-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.503583 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.503499    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.503691 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.503585    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.503691 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.503645    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.503765 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.503707    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-926d8\" (UniqueName: \"kubernetes.io/projected/dc29530c-080c-4698-a2f3-7300a77e9804-kube-api-access-926d8\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.503841 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.503822    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.503896 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.503855    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc29530c-080c-4698-a2f3-7300a77e9804-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.503950 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.503928    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc29530c-080c-4698-a2f3-7300a77e9804-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.604983 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.604949    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.605208 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.604993    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-926d8\" (UniqueName: \"kubernetes.io/projected/dc29530c-080c-4698-a2f3-7300a77e9804-kube-api-access-926d8\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.605208 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.605059    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.605208 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.605108    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc29530c-080c-4698-a2f3-7300a77e9804-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.605208 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.605160    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc29530c-080c-4698-a2f3-7300a77e9804-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.605446 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.605223    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.605446 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.605295    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc29530c-080c-4698-a2f3-7300a77e9804-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.605446 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.605344    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.605604 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.605439    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.605604 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.605531    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.605604 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.605569    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.605733 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.605677    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.605921 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.605898    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.605989 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.605968    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc29530c-080c-4698-a2f3-7300a77e9804-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.607475 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.607454    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.607562 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.607517    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc29530c-080c-4698-a2f3-7300a77e9804-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.626822 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.626799    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc29530c-080c-4698-a2f3-7300a77e9804-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.626944 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.626925    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-926d8\" (UniqueName: \"kubernetes.io/projected/dc29530c-080c-4698-a2f3-7300a77e9804-kube-api-access-926d8\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:39.717227 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:39.717196    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:40.029136 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:40.029111    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"]
Oct 30 01:12:40.031402 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:12:40.031369    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc29530c_080c_4698_a2f3_7300a77e9804.slice/crio-f5864a786d716e6091c9ee15af9731950bae7f6685e643d0a5fed967bc050aa7 WatchSource:0}: Error finding container f5864a786d716e6091c9ee15af9731950bae7f6685e643d0a5fed967bc050aa7: Status 404 returned error can't find the container with id f5864a786d716e6091c9ee15af9731950bae7f6685e643d0a5fed967bc050aa7
Oct 30 01:12:40.034270 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:40.034237    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:12:40.034359 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:40.034305    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:12:40.034359 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:40.034336    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:12:40.655300 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:40.655266    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg" event={"ID":"dc29530c-080c-4698-a2f3-7300a77e9804","Type":"ContainerStarted","Data":"21b2a38102167dbe0dd4c00c6bcdf6dc5fa3a98a1c52a1f71ba10a9ec6f74e2d"}
Oct 30 01:12:40.655700 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:40.655323    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg" event={"ID":"dc29530c-080c-4698-a2f3-7300a77e9804","Type":"ContainerStarted","Data":"f5864a786d716e6091c9ee15af9731950bae7f6685e643d0a5fed967bc050aa7"}
Oct 30 01:12:40.717391 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:40.717354    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:40.722537 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:40.722481    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg" podStartSLOduration=1.722464468 podStartE2EDuration="1.722464468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:40.707189623 +0000 UTC m=+1873.045291535" watchObservedRunningTime="2026-10-30 01:12:40.722464468 +0000 UTC m=+1873.060566379"
Oct 30 01:12:41.719728 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:41.719690    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 01:12:41.720133 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:41.719754    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg" podUID="dc29530c-080c-4698-a2f3-7300a77e9804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 01:12:42.717724 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:42.717690    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 01:12:42.717898 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:42.717746    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg" podUID="dc29530c-080c-4698-a2f3-7300a77e9804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 01:12:43.718227 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:43.718186    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 01:12:43.718602 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:43.718246    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg" podUID="dc29530c-080c-4698-a2f3-7300a77e9804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 01:12:44.717818 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:44.717782    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 01:12:44.717973 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:44.717832    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg" podUID="dc29530c-080c-4698-a2f3-7300a77e9804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 01:12:45.718015 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:45.717979    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 01:12:45.718444 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:45.718042    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg" podUID="dc29530c-080c-4698-a2f3-7300a77e9804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 01:12:46.718508 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:46.718474    4864 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 01:12:46.719010 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:46.718529    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg" podUID="dc29530c-080c-4698-a2f3-7300a77e9804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 01:12:47.724615 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:47.724577    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:47.730204 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:47.730176    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:47.743272 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:47.743242    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:12:48.642380 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:48.642341    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"]
Oct 30 01:12:48.642964 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:48.642928    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx" podUID="0967d271-ad67-457d-bd9a-de6522a24f55" containerName="maas-api" containerID="cri-o://ef93133e32a7b3ac05fb6fa124f1634bdfec5da10a658cfdcc40674b6b2a78ff" gracePeriod=30
Oct 30 01:12:48.820096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:48.816408    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx" podUID="0967d271-ad67-457d-bd9a-de6522a24f55" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.103:8443/health\": dial tcp 10.132.0.103:8443: connect: connection refused"
Oct 30 01:12:49.346018 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:12:49.345972    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:12:49.828825 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:49.828782    4864 generic.go:356] "Generic (PLEG): container finished" podID="0967d271-ad67-457d-bd9a-de6522a24f55" containerID="ef93133e32a7b3ac05fb6fa124f1634bdfec5da10a658cfdcc40674b6b2a78ff" exitCode=0
Oct 30 01:12:49.830440 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:49.829904    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx" event={"ID":"0967d271-ad67-457d-bd9a-de6522a24f55","Type":"ContainerDied","Data":"ef93133e32a7b3ac05fb6fa124f1634bdfec5da10a658cfdcc40674b6b2a78ff"}
Oct 30 01:12:49.942165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:49.942138    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"
Oct 30 01:12:50.101702 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:50.101277    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0967d271-ad67-457d-bd9a-de6522a24f55-kube-api-access-ld9rp\" (UniqueName: \"kubernetes.io/projected/0967d271-ad67-457d-bd9a-de6522a24f55-kube-api-access-ld9rp\") pod \"0967d271-ad67-457d-bd9a-de6522a24f55\" (UID: \"0967d271-ad67-457d-bd9a-de6522a24f55\") "
Oct 30 01:12:50.101702 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:50.101335    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0967d271-ad67-457d-bd9a-de6522a24f55-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0967d271-ad67-457d-bd9a-de6522a24f55-openshift-service-ca\") pod \"0967d271-ad67-457d-bd9a-de6522a24f55\" (UID: \"0967d271-ad67-457d-bd9a-de6522a24f55\") "
Oct 30 01:12:50.101702 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:50.101533    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0967d271-ad67-457d-bd9a-de6522a24f55-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0967d271-ad67-457d-bd9a-de6522a24f55-metrics-certs\") pod \"0967d271-ad67-457d-bd9a-de6522a24f55\" (UID: \"0967d271-ad67-457d-bd9a-de6522a24f55\") "
Oct 30 01:12:50.101702 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:50.101569    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0967d271-ad67-457d-bd9a-de6522a24f55-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0967d271-ad67-457d-bd9a-de6522a24f55-maas-api-tls\") pod \"0967d271-ad67-457d-bd9a-de6522a24f55\" (UID: \"0967d271-ad67-457d-bd9a-de6522a24f55\") "
Oct 30 01:12:50.102770 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:50.102732    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0967d271-ad67-457d-bd9a-de6522a24f55-openshift-service-ca" pod "0967d271-ad67-457d-bd9a-de6522a24f55" (UID: "0967d271-ad67-457d-bd9a-de6522a24f55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:50.104673 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:50.104640    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0967d271-ad67-457d-bd9a-de6522a24f55-kube-api-access-ld9rp" pod "0967d271-ad67-457d-bd9a-de6522a24f55" (UID: "0967d271-ad67-457d-bd9a-de6522a24f55"). InnerVolumeSpecName "kube-api-access-ld9rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:50.104818 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:50.104793    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0967d271-ad67-457d-bd9a-de6522a24f55-maas-api-tls" pod "0967d271-ad67-457d-bd9a-de6522a24f55" (UID: "0967d271-ad67-457d-bd9a-de6522a24f55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:50.104954 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:50.104901    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0967d271-ad67-457d-bd9a-de6522a24f55-metrics-certs" pod "0967d271-ad67-457d-bd9a-de6522a24f55" (UID: "0967d271-ad67-457d-bd9a-de6522a24f55"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:50.203097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:50.203054    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld9rp\" (UniqueName: \"kubernetes.io/projected/0967d271-ad67-457d-bd9a-de6522a24f55-kube-api-access-ld9rp\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:12:50.203097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:50.203096    4864 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0967d271-ad67-457d-bd9a-de6522a24f55-openshift-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:12:50.203350 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:50.203107    4864 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0967d271-ad67-457d-bd9a-de6522a24f55-metrics-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:12:50.203350 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:50.203116    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0967d271-ad67-457d-bd9a-de6522a24f55-maas-api-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:12:50.836595 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:50.836562    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx" event={"ID":"0967d271-ad67-457d-bd9a-de6522a24f55","Type":"ContainerDied","Data":"c0ba1f39c4a4d81b2f589c2d1db6394e18f339481bfee7ba1738bb0238dc952c"}
Oct 30 01:12:50.837126 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:50.836570    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"
Oct 30 01:12:50.837126 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:50.836659    4864 scope.go:122] "RemoveContainer" containerID="ef93133e32a7b3ac05fb6fa124f1634bdfec5da10a658cfdcc40674b6b2a78ff"
Oct 30 01:12:50.883324 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:50.883287    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"]
Oct 30 01:12:50.893510 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:50.893481    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f05249-67d98fd7c8-4qcdx"]
Oct 30 01:12:51.793510 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:51.793478    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0967d271-ad67-457d-bd9a-de6522a24f55" path="/var/lib/kubelet/pods/0967d271-ad67-457d-bd9a-de6522a24f55/volumes"
Oct 30 01:12:55.923725 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:55.923694    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl"]
Oct 30 01:12:55.924356 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:55.923950    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl" podUID="3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816" containerName="maas-api" containerID="cri-o://9a4a97427e06f49beab511cbff1b7ba7960fb2e13eef57686ae560334aee853c" gracePeriod=30
Oct 30 01:12:56.210180 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:56.210040    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl"
Oct 30 01:12:56.274736 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:56.274700    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-kube-api-access-8gzdp\" (UniqueName: \"kubernetes.io/projected/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-kube-api-access-8gzdp\") pod \"3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816\" (UID: \"3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816\") "
Oct 30 01:12:56.274916 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:56.274820    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-openshift-service-ca\") pod \"3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816\" (UID: \"3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816\") "
Oct 30 01:12:56.275111 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:56.275057    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-metrics-certs\") pod \"3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816\" (UID: \"3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816\") "
Oct 30 01:12:56.275215 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:56.275166    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-maas-api-tls\") pod \"3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816\" (UID: \"3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816\") "
Oct 30 01:12:56.275447 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:56.275408    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-openshift-service-ca" pod "3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816" (UID: "3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:56.275913 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:56.275888    4864 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-openshift-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:12:56.277836 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:56.277810    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-kube-api-access-8gzdp" pod "3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816" (UID: "3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816"). InnerVolumeSpecName "kube-api-access-8gzdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:56.277922 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:56.277819    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-metrics-certs" pod "3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816" (UID: "3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:56.277922 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:56.277838    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-maas-api-tls" pod "3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816" (UID: "3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:56.377217 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:56.377187    4864 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-metrics-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:12:56.377217 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:56.377213    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-maas-api-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:12:56.377424 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:56.377222    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gzdp\" (UniqueName: \"kubernetes.io/projected/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816-kube-api-access-8gzdp\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:12:56.874469 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:56.874430    4864 generic.go:356] "Generic (PLEG): container finished" podID="3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816" containerID="9a4a97427e06f49beab511cbff1b7ba7960fb2e13eef57686ae560334aee853c" exitCode=0
Oct 30 01:12:56.874678 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:56.874493    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl"
Oct 30 01:12:56.874678 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:56.874509    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl" event={"ID":"3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816","Type":"ContainerDied","Data":"9a4a97427e06f49beab511cbff1b7ba7960fb2e13eef57686ae560334aee853c"}
Oct 30 01:12:56.874678 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:56.874567    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl" event={"ID":"3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816","Type":"ContainerDied","Data":"a5d3a8f7003b99c22eb9397be4ea81d7a89a0b5fd42585dc9191a21e43cf7bfb"}
Oct 30 01:12:56.874678 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:56.874594    4864 scope.go:122] "RemoveContainer" containerID="9a4a97427e06f49beab511cbff1b7ba7960fb2e13eef57686ae560334aee853c"
Oct 30 01:12:56.888511 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:56.888488    4864 scope.go:122] "RemoveContainer" containerID="9a4a97427e06f49beab511cbff1b7ba7960fb2e13eef57686ae560334aee853c"
Oct 30 01:12:56.888876 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:12:56.888851    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4a97427e06f49beab511cbff1b7ba7960fb2e13eef57686ae560334aee853c\": container with ID starting with 9a4a97427e06f49beab511cbff1b7ba7960fb2e13eef57686ae560334aee853c not found: ID does not exist" containerID="9a4a97427e06f49beab511cbff1b7ba7960fb2e13eef57686ae560334aee853c"
Oct 30 01:12:56.888975 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:56.888884    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4a97427e06f49beab511cbff1b7ba7960fb2e13eef57686ae560334aee853c"} err="failed to get container status \"9a4a97427e06f49beab511cbff1b7ba7960fb2e13eef57686ae560334aee853c\": rpc error: code = NotFound desc = could not find container \"9a4a97427e06f49beab511cbff1b7ba7960fb2e13eef57686ae560334aee853c\": container with ID starting with 9a4a97427e06f49beab511cbff1b7ba7960fb2e13eef57686ae560334aee853c not found: ID does not exist"
Oct 30 01:12:56.923718 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:56.923673    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl"]
Oct 30 01:12:56.933755 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:56.933727    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1d6d47-58ff885bc-qpwwl"]
Oct 30 01:12:57.794473 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:12:57.794441    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816" path="/var/lib/kubelet/pods/3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816/volumes"
Oct 30 01:13:04.314467 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:04.314433    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"]
Oct 30 01:13:04.314970 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:04.314645    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv" podUID="29ea3935-4494-42cc-91e4-71f269edaee6" containerName="istio-proxy" containerID="cri-o://8e066bec544a9a6194cd159bbec2c877af35e0dd4191685e712d4f24e6cf69f9" gracePeriod=30
Oct 30 01:13:09.591201 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.591140    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:13:09.644742 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.644708    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29ea3935-4494-42cc-91e4-71f269edaee6-istio-token\" (UniqueName: \"kubernetes.io/projected/29ea3935-4494-42cc-91e4-71f269edaee6-istio-token\") pod \"29ea3935-4494-42cc-91e4-71f269edaee6\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") "
Oct 30 01:13:09.645062 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.644788    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-istio-envoy\") pod \"29ea3935-4494-42cc-91e4-71f269edaee6\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") "
Oct 30 01:13:09.645062 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.644838    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-workload-socket\") pod \"29ea3935-4494-42cc-91e4-71f269edaee6\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") "
Oct 30 01:13:09.645062 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.644869    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29ea3935-4494-42cc-91e4-71f269edaee6-kube-api-access-fffsm\" (UniqueName: \"kubernetes.io/projected/29ea3935-4494-42cc-91e4-71f269edaee6-kube-api-access-fffsm\") pod \"29ea3935-4494-42cc-91e4-71f269edaee6\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") "
Oct 30 01:13:09.645062 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.644914    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-workload-certs\") pod \"29ea3935-4494-42cc-91e4-71f269edaee6\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") "
Oct 30 01:13:09.645062 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.644962    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29ea3935-4494-42cc-91e4-71f269edaee6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ea3935-4494-42cc-91e4-71f269edaee6-istiod-ca-cert\") pod \"29ea3935-4494-42cc-91e4-71f269edaee6\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") "
Oct 30 01:13:09.645062 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.645001    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-istio-data\") pod \"29ea3935-4494-42cc-91e4-71f269edaee6\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") "
Oct 30 01:13:09.646939 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.645101    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29ea3935-4494-42cc-91e4-71f269edaee6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ea3935-4494-42cc-91e4-71f269edaee6-istio-podinfo\") pod \"29ea3935-4494-42cc-91e4-71f269edaee6\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") "
Oct 30 01:13:09.646939 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.645206    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-credential-socket\") pod \"29ea3935-4494-42cc-91e4-71f269edaee6\" (UID: \"29ea3935-4494-42cc-91e4-71f269edaee6\") "
Oct 30 01:13:09.646939 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.645983    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-credential-socket" pod "29ea3935-4494-42cc-91e4-71f269edaee6" (UID: "29ea3935-4494-42cc-91e4-71f269edaee6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:09.646939 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.646273    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-workload-socket" pod "29ea3935-4494-42cc-91e4-71f269edaee6" (UID: "29ea3935-4494-42cc-91e4-71f269edaee6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:09.646939 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.646302    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-workload-certs" pod "29ea3935-4494-42cc-91e4-71f269edaee6" (UID: "29ea3935-4494-42cc-91e4-71f269edaee6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:09.646939 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.646465    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-istio-data" pod "29ea3935-4494-42cc-91e4-71f269edaee6" (UID: "29ea3935-4494-42cc-91e4-71f269edaee6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:09.647665 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.647622    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29ea3935-4494-42cc-91e4-71f269edaee6-istiod-ca-cert" pod "29ea3935-4494-42cc-91e4-71f269edaee6" (UID: "29ea3935-4494-42cc-91e4-71f269edaee6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:09.649888 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.649858    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ea3935-4494-42cc-91e4-71f269edaee6-istio-token" pod "29ea3935-4494-42cc-91e4-71f269edaee6" (UID: "29ea3935-4494-42cc-91e4-71f269edaee6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:09.652770 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.652444    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29ea3935-4494-42cc-91e4-71f269edaee6-istio-podinfo" pod "29ea3935-4494-42cc-91e4-71f269edaee6" (UID: "29ea3935-4494-42cc-91e4-71f269edaee6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:13:09.652770 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.652445    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ea3935-4494-42cc-91e4-71f269edaee6-kube-api-access-fffsm" pod "29ea3935-4494-42cc-91e4-71f269edaee6" (UID: "29ea3935-4494-42cc-91e4-71f269edaee6"). InnerVolumeSpecName "kube-api-access-fffsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:09.653904 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.653882    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-istio-envoy" pod "29ea3935-4494-42cc-91e4-71f269edaee6" (UID: "29ea3935-4494-42cc-91e4-71f269edaee6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:09.746993 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.746958    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ea3935-4494-42cc-91e4-71f269edaee6-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.746993 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.746989    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.747252 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.747001    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.747252 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.747014    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fffsm\" (UniqueName: \"kubernetes.io/projected/29ea3935-4494-42cc-91e4-71f269edaee6-kube-api-access-fffsm\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.747252 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.747028    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.747252 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.747039    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ea3935-4494-42cc-91e4-71f269edaee6-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.747252 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.747048    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.747252 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.747105    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ea3935-4494-42cc-91e4-71f269edaee6-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.747252 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.747118    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ea3935-4494-42cc-91e4-71f269edaee6-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.973661 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.973628    4864 generic.go:356] "Generic (PLEG): container finished" podID="29ea3935-4494-42cc-91e4-71f269edaee6" containerID="8e066bec544a9a6194cd159bbec2c877af35e0dd4191685e712d4f24e6cf69f9" exitCode=0
Oct 30 01:13:09.973844 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.973675    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"
Oct 30 01:13:09.973844 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.973718    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv" event={"ID":"29ea3935-4494-42cc-91e4-71f269edaee6","Type":"ContainerDied","Data":"8e066bec544a9a6194cd159bbec2c877af35e0dd4191685e712d4f24e6cf69f9"}
Oct 30 01:13:09.973844 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.973780    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv" event={"ID":"29ea3935-4494-42cc-91e4-71f269edaee6","Type":"ContainerDied","Data":"c060a571389c47574be5050278b8d178af8df1b8ad03b9436a66cb8bc5b3fdd9"}
Oct 30 01:13:09.973844 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.973804    4864 scope.go:122] "RemoveContainer" containerID="8e066bec544a9a6194cd159bbec2c877af35e0dd4191685e712d4f24e6cf69f9"
Oct 30 01:13:09.983768 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.983752    4864 scope.go:122] "RemoveContainer" containerID="3efe45031f427a495fe0ee48ae19de611e42e23e7c0571108cce71e0b7667df0"
Oct 30 01:13:09.992884 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.992860    4864 scope.go:122] "RemoveContainer" containerID="8e066bec544a9a6194cd159bbec2c877af35e0dd4191685e712d4f24e6cf69f9"
Oct 30 01:13:09.993158 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:13:09.993137    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e066bec544a9a6194cd159bbec2c877af35e0dd4191685e712d4f24e6cf69f9\": container with ID starting with 8e066bec544a9a6194cd159bbec2c877af35e0dd4191685e712d4f24e6cf69f9 not found: ID does not exist" containerID="8e066bec544a9a6194cd159bbec2c877af35e0dd4191685e712d4f24e6cf69f9"
Oct 30 01:13:09.993226 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.993166    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e066bec544a9a6194cd159bbec2c877af35e0dd4191685e712d4f24e6cf69f9"} err="failed to get container status \"8e066bec544a9a6194cd159bbec2c877af35e0dd4191685e712d4f24e6cf69f9\": rpc error: code = NotFound desc = could not find container \"8e066bec544a9a6194cd159bbec2c877af35e0dd4191685e712d4f24e6cf69f9\": container with ID starting with 8e066bec544a9a6194cd159bbec2c877af35e0dd4191685e712d4f24e6cf69f9 not found: ID does not exist"
Oct 30 01:13:09.993226 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.993190    4864 scope.go:122] "RemoveContainer" containerID="3efe45031f427a495fe0ee48ae19de611e42e23e7c0571108cce71e0b7667df0"
Oct 30 01:13:09.993431 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:13:09.993413    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3efe45031f427a495fe0ee48ae19de611e42e23e7c0571108cce71e0b7667df0\": container with ID starting with 3efe45031f427a495fe0ee48ae19de611e42e23e7c0571108cce71e0b7667df0 not found: ID does not exist" containerID="3efe45031f427a495fe0ee48ae19de611e42e23e7c0571108cce71e0b7667df0"
Oct 30 01:13:09.993496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:09.993438    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3efe45031f427a495fe0ee48ae19de611e42e23e7c0571108cce71e0b7667df0"} err="failed to get container status \"3efe45031f427a495fe0ee48ae19de611e42e23e7c0571108cce71e0b7667df0\": rpc error: code = NotFound desc = could not find container \"3efe45031f427a495fe0ee48ae19de611e42e23e7c0571108cce71e0b7667df0\": container with ID starting with 3efe45031f427a495fe0ee48ae19de611e42e23e7c0571108cce71e0b7667df0 not found: ID does not exist"
Oct 30 01:13:10.031640 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:10.031609    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"]
Oct 30 01:13:10.041793 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:10.041766    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-1d6d47-openshift-default-78b47b85b5-kc7wv"]
Oct 30 01:13:11.798117 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:11.798066    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29ea3935-4494-42cc-91e4-71f269edaee6" path="/var/lib/kubelet/pods/29ea3935-4494-42cc-91e4-71f269edaee6/volumes"
Oct 30 01:13:13.539465 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.539434    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"]
Oct 30 01:13:13.540019 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.540004    4864 cpu_manager.go:414] "containerMap: removing container" podUID="3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816" containerName="maas-api"
Oct 30 01:13:13.540195 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.540020    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816" containerName="maas-api"
Oct 30 01:13:13.540195 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.540052    4864 cpu_manager.go:414] "containerMap: removing container" podUID="29ea3935-4494-42cc-91e4-71f269edaee6" containerName="istio-proxy"
Oct 30 01:13:13.540195 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.540059    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29ea3935-4494-42cc-91e4-71f269edaee6" containerName="istio-proxy"
Oct 30 01:13:13.540195 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.540097    4864 cpu_manager.go:414] "containerMap: removing container" podUID="29ea3935-4494-42cc-91e4-71f269edaee6" containerName="istio-proxy"
Oct 30 01:13:13.540195 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.540103    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29ea3935-4494-42cc-91e4-71f269edaee6" containerName="istio-proxy"
Oct 30 01:13:13.540195 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.540112    4864 cpu_manager.go:414] "containerMap: removing container" podUID="0967d271-ad67-457d-bd9a-de6522a24f55" containerName="maas-api"
Oct 30 01:13:13.540195 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.540116    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0967d271-ad67-457d-bd9a-de6522a24f55" containerName="maas-api"
Oct 30 01:13:13.540453 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.540226    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="29ea3935-4494-42cc-91e4-71f269edaee6" containerName="istio-proxy"
Oct 30 01:13:13.540453 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.540239    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="0967d271-ad67-457d-bd9a-de6522a24f55" containerName="maas-api"
Oct 30 01:13:13.540453 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.540250    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf60d4d-16e5-46c7-9dc4-d6ecad8b9816" containerName="maas-api"
Oct 30 01:13:13.540453 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.540266    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="29ea3935-4494-42cc-91e4-71f269edaee6" containerName="istio-proxy"
Oct 30 01:13:13.545255 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.545235    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.551927 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.551889    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"]
Oct 30 01:13:13.561307 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.561286    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.570599 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.570391    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"]
Oct 30 01:13:13.580337 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.580309    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"]
Oct 30 01:13:13.631101 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.631053    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.631313 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.631226    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.631313 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.631268    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.631399 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.631364    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zphp\" (UniqueName: \"kubernetes.io/projected/19edd238-c752-42b1-8d42-05c558830fdd-kube-api-access-8zphp\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.631442 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.631406    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19edd238-c752-42b1-8d42-05c558830fdd-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.631505 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.631481    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.631564 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.631550    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.631603 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.631597    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19edd238-c752-42b1-8d42-05c558830fdd-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.631640 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.631628    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19edd238-c752-42b1-8d42-05c558830fdd-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.732793 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.732754    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.732986 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.732800    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.732986 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.732834    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.732986 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.732857    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.732986 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.732895    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zphp\" (UniqueName: \"kubernetes.io/projected/19edd238-c752-42b1-8d42-05c558830fdd-kube-api-access-8zphp\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.732986 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.732973    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19edd238-c752-42b1-8d42-05c558830fdd-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.733317 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.733020    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m878p\" (UniqueName: \"kubernetes.io/projected/8e81471d-5cf3-4486-b5bf-54fb0a99702d-kube-api-access-m878p\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.733317 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.733062    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.733317 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.733208    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.733317 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.733207    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.733317 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.733289    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.733317 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.733296    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.733640 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.733401    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19edd238-c752-42b1-8d42-05c558830fdd-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.733640 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.733460    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19edd238-c752-42b1-8d42-05c558830fdd-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.733640 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.733461    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.733640 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.733492    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.733640 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.733544    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.733640 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.733605    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.733927 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.733674    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.733927 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.733716    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.733927 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.733772    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19edd238-c752-42b1-8d42-05c558830fdd-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.733927 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.733779    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.734089 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.733980    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.735462 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.735438    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.735743 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.735726    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19edd238-c752-42b1-8d42-05c558830fdd-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.754658 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.754631    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19edd238-c752-42b1-8d42-05c558830fdd-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.754787 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.754769    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zphp\" (UniqueName: \"kubernetes.io/projected/19edd238-c752-42b1-8d42-05c558830fdd-kube-api-access-8zphp\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.834868 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.834785    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m878p\" (UniqueName: \"kubernetes.io/projected/8e81471d-5cf3-4486-b5bf-54fb0a99702d-kube-api-access-m878p\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.834868 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.834828    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.835123 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.834917    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.835123 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.834936    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.835123 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.834950    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.835123 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.834977    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.835123 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.835002    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.835380 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.835117    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.835380 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.835290    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.835380 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.835338    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.835602 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.835433    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.835645 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.835631    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.835744 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.835723    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.835964 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.835948    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.837978 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.837946    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.838221 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.838199    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.856804 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.856772    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.856906 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.856876    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m878p\" (UniqueName: \"kubernetes.io/projected/8e81471d-5cf3-4486-b5bf-54fb0a99702d-kube-api-access-m878p\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:13.862639 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.862621    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:13.879434 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:13.879411    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:14.191207 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:14.191172    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"]
Oct 30 01:13:14.193603 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:13:14.193577    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19edd238_c752_42b1_8d42_05c558830fdd.slice/crio-973f758d116b332ff89a75d5a948c925161057f288e7d69cc9345b62ef215570 WatchSource:0}: Error finding container 973f758d116b332ff89a75d5a948c925161057f288e7d69cc9345b62ef215570: Status 404 returned error can't find the container with id 973f758d116b332ff89a75d5a948c925161057f288e7d69cc9345b62ef215570
Oct 30 01:13:14.194822 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:14.194802    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"]
Oct 30 01:13:14.195775 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:13:14.195754    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e81471d_5cf3_4486_b5bf_54fb0a99702d.slice/crio-376f1e969dbb92c2d4115a43173bec9419d4f18a2fed14aab9c994b6ecf83e06 WatchSource:0}: Error finding container 376f1e969dbb92c2d4115a43173bec9419d4f18a2fed14aab9c994b6ecf83e06: Status 404 returned error can't find the container with id 376f1e969dbb92c2d4115a43173bec9419d4f18a2fed14aab9c994b6ecf83e06
Oct 30 01:13:14.196270 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:14.196240    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:13:14.196362 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:14.196311    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:13:14.196484 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:14.196357    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:13:14.197944 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:14.197918    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:13:14.198034 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:14.197983    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:13:14.198128 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:14.198030    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:13:15.045869 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:15.045828    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc" event={"ID":"8e81471d-5cf3-4486-b5bf-54fb0a99702d","Type":"ContainerStarted","Data":"4d510d7637350af0e99fc4a5872eb53874e2e2b383c9cd9770fa78247c658c17"}
Oct 30 01:13:15.046357 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:15.045919    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc" event={"ID":"8e81471d-5cf3-4486-b5bf-54fb0a99702d","Type":"ContainerStarted","Data":"376f1e969dbb92c2d4115a43173bec9419d4f18a2fed14aab9c994b6ecf83e06"}
Oct 30 01:13:15.047628 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:15.047602    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6" event={"ID":"19edd238-c752-42b1-8d42-05c558830fdd","Type":"ContainerStarted","Data":"1153d402d5fcad9ed8deeef6f21da1939cd64b94cd16ee90335ee2ada1430221"}
Oct 30 01:13:15.047736 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:15.047645    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6" event={"ID":"19edd238-c752-42b1-8d42-05c558830fdd","Type":"ContainerStarted","Data":"973f758d116b332ff89a75d5a948c925161057f288e7d69cc9345b62ef215570"}
Oct 30 01:13:15.142324 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:15.142266    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc" podStartSLOduration=2.142250878 podStartE2EDuration="2.142250878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:15.140717493 +0000 UTC m=+1907.478819414" watchObservedRunningTime="2026-10-30 01:13:15.142250878 +0000 UTC m=+1907.480352788"
Oct 30 01:13:15.191097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:15.185463    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6" podStartSLOduration=2.185443732 podStartE2EDuration="2.185443732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:15.182648733 +0000 UTC m=+1907.520750702" watchObservedRunningTime="2026-10-30 01:13:15.185443732 +0000 UTC m=+1907.523545644"
Oct 30 01:13:15.877102 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:15.872158    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:15.897097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:15.896766    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:15.897097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:15.896897    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 01:13:15.897097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:15.896972    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6" podUID="19edd238-c752-42b1-8d42-05c558830fdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 01:13:15.906300 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:15.904897    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 01:13:15.906300 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:15.904974    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc" podUID="8e81471d-5cf3-4486-b5bf-54fb0a99702d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 01:13:16.876094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:16.863608    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 01:13:16.876094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:16.863678    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6" podUID="19edd238-c752-42b1-8d42-05c558830fdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 01:13:16.895096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:16.881885    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 01:13:16.895096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:16.881952    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc" podUID="8e81471d-5cf3-4486-b5bf-54fb0a99702d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 01:13:17.876098 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:17.874198    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 01:13:17.876098 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:17.874278    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6" podUID="19edd238-c752-42b1-8d42-05c558830fdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 01:13:17.881468 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:17.881434    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 01:13:17.881680 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:17.881653    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc" podUID="8e81471d-5cf3-4486-b5bf-54fb0a99702d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 01:13:18.881115 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:18.879776    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 01:13:18.881115 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:18.879870    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc" podUID="8e81471d-5cf3-4486-b5bf-54fb0a99702d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 01:13:18.881115 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:18.880682    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 01:13:18.881115 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:18.880720    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6" podUID="19edd238-c752-42b1-8d42-05c558830fdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 01:13:19.040669 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:13:19.040633    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:13:19.877096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:19.874481    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 01:13:19.877096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:19.874564    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6" podUID="19edd238-c752-42b1-8d42-05c558830fdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 01:13:19.885107 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:19.883174    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 01:13:19.885107 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:19.883230    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc" podUID="8e81471d-5cf3-4486-b5bf-54fb0a99702d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 01:13:20.863445 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:20.863404    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 01:13:20.863643 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:20.863465    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6" podUID="19edd238-c752-42b1-8d42-05c558830fdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 01:13:20.879754 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:20.879718    4864 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 01:13:20.879934 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:20.879766    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc" podUID="8e81471d-5cf3-4486-b5bf-54fb0a99702d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 01:13:21.868399 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:21.868367    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:21.868904 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:21.868763    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:21.869504 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:21.869482    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:13:21.884454 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:21.884433    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:21.884713 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:21.884691    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:21.885244 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:21.885226    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:13:30.653220 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:30.653167    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj" podUID="63a8525d-9951-4f19-b124-62e4fa91c6b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:30.726221 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:30.726172    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29" podUID="e0ab9bcb-f48a-404b-b787-9f8c43b58722" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 01:13:30.845203 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:30.845153    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll" podUID="2388a847-21ff-46dd-8b45-4183a2448cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:30.900216 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:30.900167    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh" podUID="6ba3a068-e4af-4bd1-b007-2f73f455f65c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:30.924259 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:30.924155    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv" podUID="3c545315-888c-4418-8348-b06a0080a599" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:31.230105 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:31.229426    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz" podUID="86e98e6a-b1cf-4d08-b807-26397cd1b4a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:31.496013 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:31.494828    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj" podUID="8b267884-baae-4ad3-a414-82467f7188b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:31.741977 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:31.740558    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj" podUID="63a8525d-9951-4f19-b124-62e4fa91c6b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:31.741977 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:31.740960    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29" podUID="e0ab9bcb-f48a-404b-b787-9f8c43b58722" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 01:13:31.741977 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:31.741625    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 01:13:31.741977 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:31.741684    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 01:13:31.848735 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:31.847605    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll" podUID="2388a847-21ff-46dd-8b45-4183a2448cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:31.871151 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:31.871110    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 01:13:31.908580 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:31.908538    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh" podUID="6ba3a068-e4af-4bd1-b007-2f73f455f65c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:31.914853 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:31.914191    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 01:13:31.916868 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:31.916844    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 01:13:31.917925 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:31.917900    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 01:13:31.963136 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:31.962593    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 01:13:31.990134 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:31.983539    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh"
Oct 30 01:13:40.677833 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:40.677643    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"]
Oct 30 01:13:40.678375 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:40.677913    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht" podUID="83a40d14-acf4-489f-bc5c-23449ea9c822" containerName="istio-proxy" containerID="cri-o://a683ce1fa2d2db92e7977dec3cdb193b9d4ad6821ef0a94b9ae71f6eb77df0a8" gracePeriod=30
Oct 30 01:13:45.951059 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:45.951033    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:13:46.023799 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.023761    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-workload-socket\") pod \"83a40d14-acf4-489f-bc5c-23449ea9c822\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") "
Oct 30 01:13:46.023980 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.023838    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-credential-socket\") pod \"83a40d14-acf4-489f-bc5c-23449ea9c822\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") "
Oct 30 01:13:46.024062 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.024000    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-workload-socket" pod "83a40d14-acf4-489f-bc5c-23449ea9c822" (UID: "83a40d14-acf4-489f-bc5c-23449ea9c822"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:46.024182 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.024156    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-credential-socket" pod "83a40d14-acf4-489f-bc5c-23449ea9c822" (UID: "83a40d14-acf4-489f-bc5c-23449ea9c822"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:46.024270 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.024252    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-envoy\") pod \"83a40d14-acf4-489f-bc5c-23449ea9c822\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") "
Oct 30 01:13:46.024341 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.024326    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-podinfo\") pod \"83a40d14-acf4-489f-bc5c-23449ea9c822\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") "
Oct 30 01:13:46.024399 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.024378    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-token\" (UniqueName: \"kubernetes.io/projected/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-token\") pod \"83a40d14-acf4-489f-bc5c-23449ea9c822\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") "
Oct 30 01:13:46.024459 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.024418    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-workload-certs\") pod \"83a40d14-acf4-489f-bc5c-23449ea9c822\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") "
Oct 30 01:13:46.024547 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.024515    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-data\") pod \"83a40d14-acf4-489f-bc5c-23449ea9c822\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") "
Oct 30 01:13:46.024637 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.024611    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a40d14-acf4-489f-bc5c-23449ea9c822-kube-api-access-799zp\" (UniqueName: \"kubernetes.io/projected/83a40d14-acf4-489f-bc5c-23449ea9c822-kube-api-access-799zp\") pod \"83a40d14-acf4-489f-bc5c-23449ea9c822\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") "
Oct 30 01:13:46.024718 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.024645    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83a40d14-acf4-489f-bc5c-23449ea9c822-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a40d14-acf4-489f-bc5c-23449ea9c822-istiod-ca-cert\") pod \"83a40d14-acf4-489f-bc5c-23449ea9c822\" (UID: \"83a40d14-acf4-489f-bc5c-23449ea9c822\") "
Oct 30 01:13:46.025453 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.025046    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-data" pod "83a40d14-acf4-489f-bc5c-23449ea9c822" (UID: "83a40d14-acf4-489f-bc5c-23449ea9c822"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:46.025453 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.025230    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-workload-certs" pod "83a40d14-acf4-489f-bc5c-23449ea9c822" (UID: "83a40d14-acf4-489f-bc5c-23449ea9c822"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:46.025453 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.025406    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a40d14-acf4-489f-bc5c-23449ea9c822-istiod-ca-cert" pod "83a40d14-acf4-489f-bc5c-23449ea9c822" (UID: "83a40d14-acf4-489f-bc5c-23449ea9c822"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:46.025715 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.025575    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:13:46.025715 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.025593    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:13:46.025715 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.025607    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:13:46.025715 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.025619    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:13:46.025715 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.025632    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a40d14-acf4-489f-bc5c-23449ea9c822-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:13:46.027621 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.027588    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-token" pod "83a40d14-acf4-489f-bc5c-23449ea9c822" (UID: "83a40d14-acf4-489f-bc5c-23449ea9c822"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:46.027773 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.027751    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-envoy" pod "83a40d14-acf4-489f-bc5c-23449ea9c822" (UID: "83a40d14-acf4-489f-bc5c-23449ea9c822"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:46.028164 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.028143    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a40d14-acf4-489f-bc5c-23449ea9c822-kube-api-access-799zp" pod "83a40d14-acf4-489f-bc5c-23449ea9c822" (UID: "83a40d14-acf4-489f-bc5c-23449ea9c822"). InnerVolumeSpecName "kube-api-access-799zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:46.029106 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.029067    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-podinfo" pod "83a40d14-acf4-489f-bc5c-23449ea9c822" (UID: "83a40d14-acf4-489f-bc5c-23449ea9c822"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:13:46.127371 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.127338    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:13:46.127371 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.127368    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:13:46.127371 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.127379    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a40d14-acf4-489f-bc5c-23449ea9c822-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:13:46.127611 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.127389    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-799zp\" (UniqueName: \"kubernetes.io/projected/83a40d14-acf4-489f-bc5c-23449ea9c822-kube-api-access-799zp\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:13:46.389233 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.389141    4864 generic.go:356] "Generic (PLEG): container finished" podID="83a40d14-acf4-489f-bc5c-23449ea9c822" containerID="a683ce1fa2d2db92e7977dec3cdb193b9d4ad6821ef0a94b9ae71f6eb77df0a8" exitCode=0
Oct 30 01:13:46.389233 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.389177    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht" event={"ID":"83a40d14-acf4-489f-bc5c-23449ea9c822","Type":"ContainerDied","Data":"a683ce1fa2d2db92e7977dec3cdb193b9d4ad6821ef0a94b9ae71f6eb77df0a8"}
Oct 30 01:13:46.389233 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.389203    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"
Oct 30 01:13:46.389469 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.389245    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht" event={"ID":"83a40d14-acf4-489f-bc5c-23449ea9c822","Type":"ContainerDied","Data":"7203fe472a3634fe5d1916a55118fe61ce884d938d21ff2cd44ecbb0a4136840"}
Oct 30 01:13:46.389469 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.389278    4864 scope.go:122] "RemoveContainer" containerID="a683ce1fa2d2db92e7977dec3cdb193b9d4ad6821ef0a94b9ae71f6eb77df0a8"
Oct 30 01:13:46.399577 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.399561    4864 scope.go:122] "RemoveContainer" containerID="348da3d95ff31138dd2fccd825b2213e63ad6366c220d2e9a9ae6298ab76d559"
Oct 30 01:13:46.409661 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.409643    4864 scope.go:122] "RemoveContainer" containerID="a683ce1fa2d2db92e7977dec3cdb193b9d4ad6821ef0a94b9ae71f6eb77df0a8"
Oct 30 01:13:46.409950 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:13:46.409931    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a683ce1fa2d2db92e7977dec3cdb193b9d4ad6821ef0a94b9ae71f6eb77df0a8\": container with ID starting with a683ce1fa2d2db92e7977dec3cdb193b9d4ad6821ef0a94b9ae71f6eb77df0a8 not found: ID does not exist" containerID="a683ce1fa2d2db92e7977dec3cdb193b9d4ad6821ef0a94b9ae71f6eb77df0a8"
Oct 30 01:13:46.410026 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.409963    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a683ce1fa2d2db92e7977dec3cdb193b9d4ad6821ef0a94b9ae71f6eb77df0a8"} err="failed to get container status \"a683ce1fa2d2db92e7977dec3cdb193b9d4ad6821ef0a94b9ae71f6eb77df0a8\": rpc error: code = NotFound desc = could not find container \"a683ce1fa2d2db92e7977dec3cdb193b9d4ad6821ef0a94b9ae71f6eb77df0a8\": container with ID starting with a683ce1fa2d2db92e7977dec3cdb193b9d4ad6821ef0a94b9ae71f6eb77df0a8 not found: ID does not exist"
Oct 30 01:13:46.410026 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.409991    4864 scope.go:122] "RemoveContainer" containerID="348da3d95ff31138dd2fccd825b2213e63ad6366c220d2e9a9ae6298ab76d559"
Oct 30 01:13:46.410307 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:13:46.410284    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"348da3d95ff31138dd2fccd825b2213e63ad6366c220d2e9a9ae6298ab76d559\": container with ID starting with 348da3d95ff31138dd2fccd825b2213e63ad6366c220d2e9a9ae6298ab76d559 not found: ID does not exist" containerID="348da3d95ff31138dd2fccd825b2213e63ad6366c220d2e9a9ae6298ab76d559"
Oct 30 01:13:46.410360 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.410318    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348da3d95ff31138dd2fccd825b2213e63ad6366c220d2e9a9ae6298ab76d559"} err="failed to get container status \"348da3d95ff31138dd2fccd825b2213e63ad6366c220d2e9a9ae6298ab76d559\": rpc error: code = NotFound desc = could not find container \"348da3d95ff31138dd2fccd825b2213e63ad6366c220d2e9a9ae6298ab76d559\": container with ID starting with 348da3d95ff31138dd2fccd825b2213e63ad6366c220d2e9a9ae6298ab76d559 not found: ID does not exist"
Oct 30 01:13:46.440160 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.440132    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"]
Oct 30 01:13:46.450719 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:46.450695    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f05249-openshift-default-8465ffff55-k2xht"]
Oct 30 01:13:47.793377 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:47.793342    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83a40d14-acf4-489f-bc5c-23449ea9c822" path="/var/lib/kubelet/pods/83a40d14-acf4-489f-bc5c-23449ea9c822/volumes"
Oct 30 01:13:49.043455 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:13:49.043421    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:13:53.386153 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.382364    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"]
Oct 30 01:13:53.386153 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.383151    4864 cpu_manager.go:414] "containerMap: removing container" podUID="83a40d14-acf4-489f-bc5c-23449ea9c822" containerName="istio-proxy"
Oct 30 01:13:53.386153 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.383170    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a40d14-acf4-489f-bc5c-23449ea9c822" containerName="istio-proxy"
Oct 30 01:13:53.386153 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.383203    4864 cpu_manager.go:414] "containerMap: removing container" podUID="83a40d14-acf4-489f-bc5c-23449ea9c822" containerName="istio-proxy"
Oct 30 01:13:53.386153 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.383211    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a40d14-acf4-489f-bc5c-23449ea9c822" containerName="istio-proxy"
Oct 30 01:13:53.386153 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.383361    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a40d14-acf4-489f-bc5c-23449ea9c822" containerName="istio-proxy"
Oct 30 01:13:53.386153 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.383376    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a40d14-acf4-489f-bc5c-23449ea9c822" containerName="istio-proxy"
Oct 30 01:13:53.387215 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.387190    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.405102 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.405049    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"]
Oct 30 01:13:53.428353 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.428316    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkbrj\" (UniqueName: \"kubernetes.io/projected/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-kube-api-access-tkbrj\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.428550 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.428414    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.428550 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.428442    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.428550 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.428470    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.428550 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.428498    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.428550 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.428520    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.428737 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.428556    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.428737 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.428604    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.428737 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.428620    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.529268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.529238    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.529447 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.529304    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.529447 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.529343    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.529447 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.529371    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkbrj\" (UniqueName: \"kubernetes.io/projected/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-kube-api-access-tkbrj\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.529596 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.529500    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.529596 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.529526    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.529596 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.529574    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.529771 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.529619    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.529771 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.529659    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.529771 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.529699    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.529956 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.529929    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.530032 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.529949    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.530032 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.529990    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.530728 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.530703    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.532009 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.531986    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.532122 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.531986    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.551520 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.551488    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.551872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.551607    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkbrj\" (UniqueName: \"kubernetes.io/projected/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-kube-api-access-tkbrj\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:53.709456 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:53.709425    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:54.030804 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:54.030775    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"]
Oct 30 01:13:54.032635 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:13:54.032608    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ebf7d4_a370_441f_88c3_10c9ff7ee772.slice/crio-b737380a6ec10440329dc5dade1a32a5c3c7818009e997baa3e93d9003c7bb9d WatchSource:0}: Error finding container b737380a6ec10440329dc5dade1a32a5c3c7818009e997baa3e93d9003c7bb9d: Status 404 returned error can't find the container with id b737380a6ec10440329dc5dade1a32a5c3c7818009e997baa3e93d9003c7bb9d
Oct 30 01:13:54.035087 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:54.035040    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:13:54.035189 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:54.035134    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:13:54.035189 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:54.035165    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:13:54.452388 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:54.452348    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54" event={"ID":"a4ebf7d4-a370-441f-88c3-10c9ff7ee772","Type":"ContainerStarted","Data":"2325cabd837a18ecc0447f0c4aff7cf71a0cb8d1d80608c6ad329f45a3fbff73"}
Oct 30 01:13:54.452875 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:54.452424    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54" event={"ID":"a4ebf7d4-a370-441f-88c3-10c9ff7ee772","Type":"ContainerStarted","Data":"b737380a6ec10440329dc5dade1a32a5c3c7818009e997baa3e93d9003c7bb9d"}
Oct 30 01:13:54.508014 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:54.507947    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54" podStartSLOduration=1.507926421 podStartE2EDuration="1.507926421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:54.504439168 +0000 UTC m=+1946.842541116" watchObservedRunningTime="2026-10-30 01:13:54.507926421 +0000 UTC m=+1946.846028328"
Oct 30 01:13:54.710360 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:54.710212    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:13:55.713430 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:55.713390    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 01:13:55.714028 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:55.713995    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54" podUID="a4ebf7d4-a370-441f-88c3-10c9ff7ee772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 01:13:56.710129 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:56.710057    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 01:13:56.710313 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:56.710178    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54" podUID="a4ebf7d4-a370-441f-88c3-10c9ff7ee772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 01:13:57.710097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:57.710043    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 01:13:57.710503 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:57.710125    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54" podUID="a4ebf7d4-a370-441f-88c3-10c9ff7ee772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 01:13:58.710548 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:58.710504    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 01:13:58.710970 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:58.710577    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54" podUID="a4ebf7d4-a370-441f-88c3-10c9ff7ee772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 01:13:59.710856 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:59.710817    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 01:13:59.711420 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:13:59.710876    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54" podUID="a4ebf7d4-a370-441f-88c3-10c9ff7ee772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 01:14:00.710763 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:00.710718    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 01:14:00.710951 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:00.710814    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54" podUID="a4ebf7d4-a370-441f-88c3-10c9ff7ee772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 01:14:02.032986 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:02.032955    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:14:02.033519 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:02.033496    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:14:02.036018 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:02.035995    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:14:18.360487 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.360449    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n"]
Oct 30 01:14:18.364636 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.364612    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n"
Oct 30 01:14:18.386890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.386855    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n"]
Oct 30 01:14:18.392996 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.392968    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-df4db1-dockercfg-2ztgw\""
Oct 30 01:14:18.435712 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.435681    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-df4db1\""
Oct 30 01:14:18.478013 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.477977    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0822059-3672-4595-967c-0796222fd966-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n\" (UID: \"d0822059-3672-4595-967c-0796222fd966\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n"
Oct 30 01:14:18.478233 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.478179    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shkq9\" (UniqueName: \"kubernetes.io/projected/d0822059-3672-4595-967c-0796222fd966-kube-api-access-shkq9\") pod \"payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n\" (UID: \"d0822059-3672-4595-967c-0796222fd966\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n"
Oct 30 01:14:18.581317 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.581269    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0822059-3672-4595-967c-0796222fd966-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n\" (UID: \"d0822059-3672-4595-967c-0796222fd966\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n"
Oct 30 01:14:18.581825 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.581792    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shkq9\" (UniqueName: \"kubernetes.io/projected/d0822059-3672-4595-967c-0796222fd966-kube-api-access-shkq9\") pod \"payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n\" (UID: \"d0822059-3672-4595-967c-0796222fd966\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n"
Oct 30 01:14:18.584099 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.584052    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0822059-3672-4595-967c-0796222fd966-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n\" (UID: \"d0822059-3672-4595-967c-0796222fd966\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n"
Oct 30 01:14:18.605108 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.605055    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shkq9\" (UniqueName: \"kubernetes.io/projected/d0822059-3672-4595-967c-0796222fd966-kube-api-access-shkq9\") pod \"payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n\" (UID: \"d0822059-3672-4595-967c-0796222fd966\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n"
Oct 30 01:14:18.679376 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.679290    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n"
Oct 30 01:14:18.787554 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.787516    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"]
Oct 30 01:14:18.798105 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.792804    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"
Oct 30 01:14:18.806867 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.806842    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-0e1a98-serving-cert\""
Oct 30 01:14:18.814114 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.814085    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"]
Oct 30 01:14:18.885890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.885855    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khmms\" (UniqueName: \"kubernetes.io/projected/9f728c20-eae2-4fd4-aa40-4094a75f5f16-kube-api-access-khmms\") pod \"maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f\" (UID: \"9f728c20-eae2-4fd4-aa40-4094a75f5f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"
Oct 30 01:14:18.886105 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.885987    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f728c20-eae2-4fd4-aa40-4094a75f5f16-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f\" (UID: \"9f728c20-eae2-4fd4-aa40-4094a75f5f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"
Oct 30 01:14:18.886191 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.886159    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f728c20-eae2-4fd4-aa40-4094a75f5f16-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f\" (UID: \"9f728c20-eae2-4fd4-aa40-4094a75f5f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"
Oct 30 01:14:18.886359 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.886340    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f728c20-eae2-4fd4-aa40-4094a75f5f16-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f\" (UID: \"9f728c20-eae2-4fd4-aa40-4094a75f5f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"
Oct 30 01:14:18.983188 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.983163    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n"]
Oct 30 01:14:18.985142 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:14:18.985111    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0822059_3672_4595_967c_0796222fd966.slice/crio-f87d57e1ce1043cb49d2b6100e2157aaf97a68f2073b2adaa9e2a435744299bc WatchSource:0}: Error finding container f87d57e1ce1043cb49d2b6100e2157aaf97a68f2073b2adaa9e2a435744299bc: Status 404 returned error can't find the container with id f87d57e1ce1043cb49d2b6100e2157aaf97a68f2073b2adaa9e2a435744299bc
Oct 30 01:14:18.987138 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.987111    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f728c20-eae2-4fd4-aa40-4094a75f5f16-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f\" (UID: \"9f728c20-eae2-4fd4-aa40-4094a75f5f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"
Oct 30 01:14:18.987244 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.987165    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f728c20-eae2-4fd4-aa40-4094a75f5f16-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f\" (UID: \"9f728c20-eae2-4fd4-aa40-4094a75f5f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"
Oct 30 01:14:18.987244 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.987227    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khmms\" (UniqueName: \"kubernetes.io/projected/9f728c20-eae2-4fd4-aa40-4094a75f5f16-kube-api-access-khmms\") pod \"maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f\" (UID: \"9f728c20-eae2-4fd4-aa40-4094a75f5f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"
Oct 30 01:14:18.987344 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.987320    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f728c20-eae2-4fd4-aa40-4094a75f5f16-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f\" (UID: \"9f728c20-eae2-4fd4-aa40-4094a75f5f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"
Oct 30 01:14:18.987573 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:14:18.987544    4864 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-serving-cert: secret "maas-api-e2e-shared-a-w1-0e1a98-serving-cert" not found
Oct 30 01:14:18.987698 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:14:18.987631    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f728c20-eae2-4fd4-aa40-4094a75f5f16-maas-api-tls podName:9f728c20-eae2-4fd4-aa40-4094a75f5f16 nodeName:}" failed. No retries permitted until 2026-10-30 01:14:19.487611611 +0000 UTC m=+1971.825713523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f728c20-eae2-4fd4-aa40-4094a75f5f16-maas-api-tls") pod "maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f" (UID: "9f728c20-eae2-4fd4-aa40-4094a75f5f16") : secret "maas-api-e2e-shared-a-w1-0e1a98-serving-cert" not found
Oct 30 01:14:18.987862 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.987839    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f728c20-eae2-4fd4-aa40-4094a75f5f16-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f\" (UID: \"9f728c20-eae2-4fd4-aa40-4094a75f5f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"
Oct 30 01:14:18.990041 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:18.990006    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f728c20-eae2-4fd4-aa40-4094a75f5f16-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f\" (UID: \"9f728c20-eae2-4fd4-aa40-4094a75f5f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"
Oct 30 01:14:19.022034 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:19.022003    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khmms\" (UniqueName: \"kubernetes.io/projected/9f728c20-eae2-4fd4-aa40-4094a75f5f16-kube-api-access-khmms\") pod \"maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f\" (UID: \"9f728c20-eae2-4fd4-aa40-4094a75f5f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"
Oct 30 01:14:19.082251 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:14:19.082203    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:14:19.494057 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:19.494021    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f728c20-eae2-4fd4-aa40-4094a75f5f16-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f\" (UID: \"9f728c20-eae2-4fd4-aa40-4094a75f5f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"
Oct 30 01:14:19.496662 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:19.496631    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f728c20-eae2-4fd4-aa40-4094a75f5f16-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f\" (UID: \"9f728c20-eae2-4fd4-aa40-4094a75f5f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"
Oct 30 01:14:19.662993 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:19.662908    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n" event={"ID":"d0822059-3672-4595-967c-0796222fd966","Type":"ContainerStarted","Data":"b90616317662e8bc1eaa935b9f27ae9a6a54e8b9035aa164a9c384f3c83b9a2c"}
Oct 30 01:14:19.663211 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:19.663011    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n" event={"ID":"d0822059-3672-4595-967c-0796222fd966","Type":"ContainerStarted","Data":"f87d57e1ce1043cb49d2b6100e2157aaf97a68f2073b2adaa9e2a435744299bc"}
Oct 30 01:14:19.663315 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:19.663293    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n"
Oct 30 01:14:19.704152 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:19.704087    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n" podStartSLOduration=1.7040540370000001 podStartE2EDuration="1.704054037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:19.703087548 +0000 UTC m=+1972.041189452" watchObservedRunningTime="2026-10-30 01:14:19.704054037 +0000 UTC m=+1972.042155951"
Oct 30 01:14:19.720506 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:19.720479    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"
Oct 30 01:14:20.049803 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:20.049682    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"]
Oct 30 01:14:20.108389 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:20.108360    4864 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:14:20.704466 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:20.704430    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f" event={"ID":"9f728c20-eae2-4fd4-aa40-4094a75f5f16","Type":"ContainerStarted","Data":"5c123d2e2c324915d0334e534e27bbc6af5548c3a8b925070d85e68635cd8240"}
Oct 30 01:14:20.705352 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:20.705326    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f" event={"ID":"9f728c20-eae2-4fd4-aa40-4094a75f5f16","Type":"ContainerStarted","Data":"3dc32146bd25d14d0fa50285e3d64994aa8c70aef448ce0ebdfdfaa44cc68bed"}
Oct 30 01:14:20.705514 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:20.705498    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"
Oct 30 01:14:27.728813 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:27.728784    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"
Oct 30 01:14:27.770647 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:27.770591    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f" podStartSLOduration=9.365638212 podStartE2EDuration="9.770576777s" totalImagesPullingTime="404.938565ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:20.747780011 +0000 UTC m=+1973.085881922" watchObservedRunningTime="2026-10-30 01:14:27.770576777 +0000 UTC m=+1980.108678686"
Oct 30 01:14:30.707652 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:30.707623    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n"
Oct 30 01:14:38.513639 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.513601    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"]
Oct 30 01:14:38.519180 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.519158    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.545482 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.545441    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"]
Oct 30 01:14:38.638311 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.638267    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.638311 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.638315    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swjvn\" (UniqueName: \"kubernetes.io/projected/0838696e-e4dd-4f9f-90b8-579c9955ac74-kube-api-access-swjvn\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.638585 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.638407    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.638585 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.638453    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.638585 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.638541    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.638585 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.638581    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.638760 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.638687    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.638760 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.638744    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0838696e-e4dd-4f9f-90b8-579c9955ac74-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.638866 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.638834    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.740429 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.740393    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.740625 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.740484    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0838696e-e4dd-4f9f-90b8-579c9955ac74-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.740625 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.740562    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.740745 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.740713    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.740805 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.740742    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swjvn\" (UniqueName: \"kubernetes.io/projected/0838696e-e4dd-4f9f-90b8-579c9955ac74-kube-api-access-swjvn\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.740805 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.740793    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.740915 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.740836    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.740915 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.740870    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.740915 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.740902    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.741086 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.740926    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.741197 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.741177    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.741313 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.741291    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.741408 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.741389    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.741548 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.741528    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0838696e-e4dd-4f9f-90b8-579c9955ac74-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.742823 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.742805    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.743292 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.743272    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.762291 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.762257    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.762469 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.762455    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swjvn\" (UniqueName: \"kubernetes.io/projected/0838696e-e4dd-4f9f-90b8-579c9955ac74-kube-api-access-swjvn\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:38.833038 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:38.832944    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:39.154912 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:39.154879    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"]
Oct 30 01:14:39.155903 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:14:39.155875    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0838696e_e4dd_4f9f_90b8_579c9955ac74.slice/crio-64ef7e2da7ff83e1c282e622c902a9b45ec75d31408a3c2c84c42022e9af3dd7 WatchSource:0}: Error finding container 64ef7e2da7ff83e1c282e622c902a9b45ec75d31408a3c2c84c42022e9af3dd7: Status 404 returned error can't find the container with id 64ef7e2da7ff83e1c282e622c902a9b45ec75d31408a3c2c84c42022e9af3dd7
Oct 30 01:14:39.158378 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:39.158347    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:14:39.158457 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:39.158412    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:14:39.158457 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:39.158451    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:14:39.841892 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:39.841851    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j" event={"ID":"0838696e-e4dd-4f9f-90b8-579c9955ac74","Type":"ContainerStarted","Data":"09db19c32d2d80a19eaf88774e9d04ce9e23b569a63572a941fa669261839f17"}
Oct 30 01:14:39.842624 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:39.842597    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j" event={"ID":"0838696e-e4dd-4f9f-90b8-579c9955ac74","Type":"ContainerStarted","Data":"64ef7e2da7ff83e1c282e622c902a9b45ec75d31408a3c2c84c42022e9af3dd7"}
Oct 30 01:14:39.899576 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:39.899511    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j" podStartSLOduration=1.899489625 podStartE2EDuration="1.899489625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:39.894848707 +0000 UTC m=+1992.232950628" watchObservedRunningTime="2026-10-30 01:14:39.899489625 +0000 UTC m=+1992.237591535"
Oct 30 01:14:40.833924 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:40.833884    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:40.835340 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:40.835315    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 01:14:40.835457 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:40.835367    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j" podUID="0838696e-e4dd-4f9f-90b8-579c9955ac74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 01:14:41.833973 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:41.833939    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 01:14:41.834546 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:41.834056    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j" podUID="0838696e-e4dd-4f9f-90b8-579c9955ac74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 01:14:42.833979 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:42.833945    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 01:14:42.834485 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:42.834460    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j" podUID="0838696e-e4dd-4f9f-90b8-579c9955ac74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 01:14:43.834438 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:43.834396    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 01:14:43.834862 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:43.834461    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j" podUID="0838696e-e4dd-4f9f-90b8-579c9955ac74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 01:14:44.837519 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:44.837480    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 01:14:44.838164 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:44.838133    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j" podUID="0838696e-e4dd-4f9f-90b8-579c9955ac74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 01:14:45.407417 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.407382    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"]
Oct 30 01:14:45.415491 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.415469    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.430057 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.430031    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-1854c4-openshift-default-dockercfg-c8skz\""
Oct 30 01:14:45.445655 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.445622    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"]
Oct 30 01:14:45.487230 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.487194    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7bwr\" (UniqueName: \"kubernetes.io/projected/3a94e905-9533-4db9-b8d2-5c10538e0d0b-kube-api-access-z7bwr\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.487418 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.487248    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-workload-certs\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.487418 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.487376    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-workload-socket\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.487524 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.487419    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-envoy\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.487524 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.487439    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-data\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.487624 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.487605    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-podinfo\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.487660 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.487650    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-credential-socket\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.487693 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.487667    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-token\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.487743 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.487698    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istiod-ca-cert\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.588904 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.588871    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-podinfo\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.589121 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.588946    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-credential-socket\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.589502 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.589479    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-credential-socket\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.589909 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.589889    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-token\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.590056 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.590039    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istiod-ca-cert\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.590493 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.590382    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7bwr\" (UniqueName: \"kubernetes.io/projected/3a94e905-9533-4db9-b8d2-5c10538e0d0b-kube-api-access-z7bwr\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.590878 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.590858    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istiod-ca-cert\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.591028 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.591010    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-workload-certs\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.591144 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.591052    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-workload-certs\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.591398 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.591381    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-workload-socket\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.591657 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.591640    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-workload-socket\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.591751 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.591718    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-envoy\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.591812 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.591752    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-data\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.592313 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.592295    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-data\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.593680 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.593656    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-podinfo\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.594561 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.594538    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-envoy\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.613794 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.613766    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-token\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.614122 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.614102    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7bwr\" (UniqueName: \"kubernetes.io/projected/3a94e905-9533-4db9-b8d2-5c10538e0d0b-kube-api-access-z7bwr\") pod \"e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") " pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.730665 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.730625    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:45.838979 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.838605    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 01:14:45.838979 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:45.838652    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j" podUID="0838696e-e4dd-4f9f-90b8-579c9955ac74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 01:14:46.055174 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:46.055146    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"]
Oct 30 01:14:46.056960 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:14:46.056931    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a94e905_9533_4db9_b8d2_5c10538e0d0b.slice/crio-5d70542fe175ea94442c43fa30454e69dc5bd33705fc2c1800f0ba8426115792 WatchSource:0}: Error finding container 5d70542fe175ea94442c43fa30454e69dc5bd33705fc2c1800f0ba8426115792: Status 404 returned error can't find the container with id 5d70542fe175ea94442c43fa30454e69dc5bd33705fc2c1800f0ba8426115792
Oct 30 01:14:46.059836 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:46.059801    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:14:46.059971 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:46.059867    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:14:46.059971 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:46.059896    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:14:46.927482 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:46.927438    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk" event={"ID":"3a94e905-9533-4db9-b8d2-5c10538e0d0b","Type":"ContainerStarted","Data":"e9406d9dc7f85a9a274b160d43844a46acf193bd2ed252ccb1428d25316824e2"}
Oct 30 01:14:46.927988 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:46.927544    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk" event={"ID":"3a94e905-9533-4db9-b8d2-5c10538e0d0b","Type":"ContainerStarted","Data":"5d70542fe175ea94442c43fa30454e69dc5bd33705fc2c1800f0ba8426115792"}
Oct 30 01:14:46.935658 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:46.935629    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:46.935951 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:46.935924    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:46.936657 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:46.936635    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:14:46.979760 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:46.979708    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk" podStartSLOduration=1.979672942 podStartE2EDuration="1.979672942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:46.978298345 +0000 UTC m=+1999.316400270" watchObservedRunningTime="2026-10-30 01:14:46.979672942 +0000 UTC m=+1999.317774852"
Oct 30 01:14:47.731760 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:47.731729    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:47.736838 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:47.736809    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:47.934840 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:47.934811    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:47.935619 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:47.935599    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:14:49.040734 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:14:49.040698    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:14:56.787165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:56.787130    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"]
Oct 30 01:14:56.792106 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:56.792082    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:56.806807 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:56.806783    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-df4db1-openshift-default-dockercfg-49p7w\""
Oct 30 01:14:56.820791 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:56.820764    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"]
Oct 30 01:14:56.899023 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:56.898950    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c5848bd-c585-46df-93d6-d63b46d80c3b-istiod-ca-cert\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:56.901206 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:56.901175    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-podinfo\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:56.901496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:56.901469    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-workload-certs\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:56.901716 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:56.901695    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-data\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:56.901899 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:56.901877    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dphvb\" (UniqueName: \"kubernetes.io/projected/2c5848bd-c585-46df-93d6-d63b46d80c3b-kube-api-access-dphvb\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:56.902229 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:56.902201    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-workload-socket\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:56.902451 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:56.902422    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-credential-socket\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:56.903781 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:56.903758    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-token\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:56.903991 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:56.903974    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-envoy\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:57.015267 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:57.008539    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-podinfo\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:57.015267 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:57.008597    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-workload-certs\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:57.015267 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:57.008665    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-data\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:57.015267 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:57.008696    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dphvb\" (UniqueName: \"kubernetes.io/projected/2c5848bd-c585-46df-93d6-d63b46d80c3b-kube-api-access-dphvb\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:57.015267 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:57.008822    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-workload-socket\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:57.015267 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:57.008848    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-credential-socket\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:57.015267 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:57.008875    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-token\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:57.015267 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:57.008942    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-envoy\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:57.015267 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:57.009021    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c5848bd-c585-46df-93d6-d63b46d80c3b-istiod-ca-cert\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:57.015267 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:57.009993    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c5848bd-c585-46df-93d6-d63b46d80c3b-istiod-ca-cert\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:57.031284 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:57.026882    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-workload-socket\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:57.058876 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:57.058793    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-credential-socket\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:57.128909 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:57.067323    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-data\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:57.128909 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:57.067558    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-workload-certs\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:57.128909 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:57.091784    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-envoy\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:57.128909 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:57.095941    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dphvb\" (UniqueName: \"kubernetes.io/projected/2c5848bd-c585-46df-93d6-d63b46d80c3b-kube-api-access-dphvb\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:57.128909 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:57.099374    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-token\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:57.135751 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:57.135722    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-podinfo\") pod \"e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") " pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:57.406365 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:57.406286    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:57.782529 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:57.782499    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"]
Oct 30 01:14:57.784524 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:14:57.784490    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c5848bd_c585_46df_93d6_d63b46d80c3b.slice/crio-a68158840a95b632f75fdbcde739dce2662646bcdb75b74f909d252898e4550e WatchSource:0}: Error finding container a68158840a95b632f75fdbcde739dce2662646bcdb75b74f909d252898e4550e: Status 404 returned error can't find the container with id a68158840a95b632f75fdbcde739dce2662646bcdb75b74f909d252898e4550e
Oct 30 01:14:57.787383 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:57.787348    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:14:57.787755 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:57.787416    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:14:57.787755 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:57.787460    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:14:58.121162 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:58.120197    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52" event={"ID":"2c5848bd-c585-46df-93d6-d63b46d80c3b","Type":"ContainerStarted","Data":"bea4092a4088e53dcf2734de2f0dd0d8a5656f94f75e4b7fbef91d78e6b23c99"}
Oct 30 01:14:58.121162 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:58.120279    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52" event={"ID":"2c5848bd-c585-46df-93d6-d63b46d80c3b","Type":"ContainerStarted","Data":"a68158840a95b632f75fdbcde739dce2662646bcdb75b74f909d252898e4550e"}
Oct 30 01:14:58.207095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:58.194492    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52" podStartSLOduration=2.194472267 podStartE2EDuration="2.194472267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:58.176231531 +0000 UTC m=+2010.514333444" watchObservedRunningTime="2026-10-30 01:14:58.194472267 +0000 UTC m=+2010.532574182"
Oct 30 01:14:58.407356 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:58.407266    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:14:58.408722 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:58.408685    4864 patch_prober.go:29] interesting pod/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 01:14:58.408837 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:58.408745    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52" podUID="2c5848bd-c585-46df-93d6-d63b46d80c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 01:14:59.407163 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:59.407126    4864 patch_prober.go:29] interesting pod/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 01:14:59.407552 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:59.407179    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52" podUID="2c5848bd-c585-46df-93d6-d63b46d80c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 01:14:59.989114 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:59.989059    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"]
Oct 30 01:14:59.994349 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:14:59.994322    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.009961 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.009924    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"]
Oct 30 01:15:00.015107 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.015041    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-0e1a98-openshift-default-dockercfg-tlhv9\""
Oct 30 01:15:00.102197 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.102156    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-workload-socket\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.102406 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.102239    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-workload-certs\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.102406 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.102276    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-podinfo\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.102406 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.102295    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istiod-ca-cert\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.102591 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.102444    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-credential-socket\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.102591 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.102472    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-data\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.102591 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.102491    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-token\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.102767 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.102604    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-envoy\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.102824 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.102760    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-644m5\" (UniqueName: \"kubernetes.io/projected/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-kube-api-access-644m5\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.209092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.204465    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-644m5\" (UniqueName: \"kubernetes.io/projected/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-kube-api-access-644m5\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.209092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.204550    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-workload-socket\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.209092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.204607    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-workload-certs\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.209092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.204649    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-podinfo\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.209092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.204681    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istiod-ca-cert\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.209092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.204776    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-credential-socket\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.209092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.204800    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-data\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.209092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.204826    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-token\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.209092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.204922    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-envoy\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.212355 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.210920    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-workload-socket\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.212355 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.211514    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-workload-certs\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.212355 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.211743    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-credential-socket\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.230142 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.228531    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-data\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.230142 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.228797    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-podinfo\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.230397 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.230375    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istiod-ca-cert\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.267559 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.255488    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-644m5\" (UniqueName: \"kubernetes.io/projected/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-kube-api-access-644m5\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.267559 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.255953    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-envoy\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.267559 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.256037    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-token\") pod \"e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") " pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.314675 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.314645    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:00.409031 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.409002    4864 patch_prober.go:29] interesting pod/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 01:15:00.409605 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.409572    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52" podUID="2c5848bd-c585-46df-93d6-d63b46d80c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 01:15:00.683838 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.683738    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"]
Oct 30 01:15:00.687685 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:15:00.687657    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11b8ed4_c24c_4919_959f_86cf8ecbacb2.slice/crio-f1105dedad5826040d7fbeb6cdba266762456107256e204279511121ec7b78a7 WatchSource:0}: Error finding container f1105dedad5826040d7fbeb6cdba266762456107256e204279511121ec7b78a7: Status 404 returned error can't find the container with id f1105dedad5826040d7fbeb6cdba266762456107256e204279511121ec7b78a7
Oct 30 01:15:00.690709 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.690675    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:15:00.690819 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.690748    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:15:00.690819 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:00.690781    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:15:01.144697 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:01.144663    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8" event={"ID":"a11b8ed4-c24c-4919-959f-86cf8ecbacb2","Type":"ContainerStarted","Data":"1a40f0ec4a538e39c27aa6141ed7e275d190a56bed567a5486f160814619efeb"}
Oct 30 01:15:01.144872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:01.144717    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8" event={"ID":"a11b8ed4-c24c-4919-959f-86cf8ecbacb2","Type":"ContainerStarted","Data":"f1105dedad5826040d7fbeb6cdba266762456107256e204279511121ec7b78a7"}
Oct 30 01:15:01.191629 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:01.191562    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8" podStartSLOduration=2.191545519 podStartE2EDuration="2.191545519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:01.19060488 +0000 UTC m=+2013.528706791" watchObservedRunningTime="2026-10-30 01:15:01.191545519 +0000 UTC m=+2013.529647429"
Oct 30 01:15:01.316848 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:01.316322    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:01.317149 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:01.317124    4864 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 01:15:01.317391 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:01.317264    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8" podUID="a11b8ed4-c24c-4919-959f-86cf8ecbacb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 01:15:01.408204 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:01.407470    4864 patch_prober.go:29] interesting pod/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 01:15:01.408204 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:01.407531    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52" podUID="2c5848bd-c585-46df-93d6-d63b46d80c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 01:15:02.346098 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:02.328205    4864 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 01:15:02.346098 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:02.328268    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8" podUID="a11b8ed4-c24c-4919-959f-86cf8ecbacb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 01:15:02.409094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:02.408428    4864 patch_prober.go:29] interesting pod/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 01:15:02.409094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:02.408488    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52" podUID="2c5848bd-c585-46df-93d6-d63b46d80c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 01:15:03.315560 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:03.315523    4864 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 01:15:03.315767 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:03.315671    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8" podUID="a11b8ed4-c24c-4919-959f-86cf8ecbacb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 01:15:03.408092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:03.407621    4864 patch_prober.go:29] interesting pod/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 01:15:03.408092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:03.407701    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52" podUID="2c5848bd-c585-46df-93d6-d63b46d80c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 01:15:04.315851 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:04.315818    4864 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 01:15:04.316024 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:04.315870    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8" podUID="a11b8ed4-c24c-4919-959f-86cf8ecbacb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 01:15:04.411183 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:04.411156    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:15:04.411934 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:04.411913    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:15:04.412618 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:04.412600    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:15:05.316489 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:05.316448    4864 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 01:15:05.316684 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:05.316520    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8" podUID="a11b8ed4-c24c-4919-959f-86cf8ecbacb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 01:15:06.315777 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:06.315747    4864 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 01:15:06.316200 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:06.315800    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8" podUID="a11b8ed4-c24c-4919-959f-86cf8ecbacb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 01:15:07.321121 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:07.321091    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:07.321544 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:07.321533    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:07.322267 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:07.322246    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:15:19.040706 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:15:19.040626    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:15:36.366165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:36.366129    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl"]
Oct 30 01:15:36.372001 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:36.371975    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl"
Oct 30 01:15:36.388306 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:36.388275    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-9da49f-serving-cert\""
Oct 30 01:15:36.409252 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:36.409220    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl"]
Oct 30 01:15:36.446916 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:36.446872    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl\" (UID: \"79ebbfb0-bf32-49c1-8feb-60b55eadbf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl"
Oct 30 01:15:36.447111 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:36.447030    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl\" (UID: \"79ebbfb0-bf32-49c1-8feb-60b55eadbf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl"
Oct 30 01:15:36.447239 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:36.447215    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl\" (UID: \"79ebbfb0-bf32-49c1-8feb-60b55eadbf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl"
Oct 30 01:15:36.447294 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:36.447284    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxcxj\" (UniqueName: \"kubernetes.io/projected/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-kube-api-access-kxcxj\") pod \"maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl\" (UID: \"79ebbfb0-bf32-49c1-8feb-60b55eadbf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl"
Oct 30 01:15:36.548219 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:36.548179    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl\" (UID: \"79ebbfb0-bf32-49c1-8feb-60b55eadbf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl"
Oct 30 01:15:36.548426 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:36.548243    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxcxj\" (UniqueName: \"kubernetes.io/projected/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-kube-api-access-kxcxj\") pod \"maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl\" (UID: \"79ebbfb0-bf32-49c1-8feb-60b55eadbf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl"
Oct 30 01:15:36.548426 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:36.548388    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl\" (UID: \"79ebbfb0-bf32-49c1-8feb-60b55eadbf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl"
Oct 30 01:15:36.548558 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:36.548480    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl\" (UID: \"79ebbfb0-bf32-49c1-8feb-60b55eadbf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl"
Oct 30 01:15:36.548952 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:36.548925    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl\" (UID: \"79ebbfb0-bf32-49c1-8feb-60b55eadbf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl"
Oct 30 01:15:36.551256 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:36.551227    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl\" (UID: \"79ebbfb0-bf32-49c1-8feb-60b55eadbf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl"
Oct 30 01:15:36.551381 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:36.551264    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl\" (UID: \"79ebbfb0-bf32-49c1-8feb-60b55eadbf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl"
Oct 30 01:15:36.570668 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:36.570632    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxcxj\" (UniqueName: \"kubernetes.io/projected/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-kube-api-access-kxcxj\") pod \"maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl\" (UID: \"79ebbfb0-bf32-49c1-8feb-60b55eadbf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl"
Oct 30 01:15:36.688027 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:36.687908    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl"
Oct 30 01:15:37.025766 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:37.025741    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl"]
Oct 30 01:15:37.027743 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:15:37.027718    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ebbfb0_bf32_49c1_8feb_60b55eadbf55.slice/crio-e20aa467724e3956dfb085989ccc540ddf0e9786625a3cacd32518bc027a355d WatchSource:0}: Error finding container e20aa467724e3956dfb085989ccc540ddf0e9786625a3cacd32518bc027a355d: Status 404 returned error can't find the container with id e20aa467724e3956dfb085989ccc540ddf0e9786625a3cacd32518bc027a355d
Oct 30 01:15:37.520319 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:37.520273    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl" event={"ID":"79ebbfb0-bf32-49c1-8feb-60b55eadbf55","Type":"ContainerStarted","Data":"58fc5a522e2fc68e2a088ceaacfcf6ed8d5ad4ab72d28aa0cfd9937338cdb14e"}
Oct 30 01:15:37.520766 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:37.520360    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl"
Oct 30 01:15:37.520766 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:37.520388    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl" event={"ID":"79ebbfb0-bf32-49c1-8feb-60b55eadbf55","Type":"ContainerStarted","Data":"e20aa467724e3956dfb085989ccc540ddf0e9786625a3cacd32518bc027a355d"}
Oct 30 01:15:37.571946 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:37.571888    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl" podStartSLOduration=1.206382912 podStartE2EDuration="1.571869682s" totalImagesPullingTime="365.48677ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:37.569331466 +0000 UTC m=+2049.907433375" watchObservedRunningTime="2026-10-30 01:15:37.571869682 +0000 UTC m=+2049.909971591"
Oct 30 01:15:39.536231 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:39.536194    4864 generic.go:356] "Generic (PLEG): container finished" podID="3a94e905-9533-4db9-b8d2-5c10538e0d0b" containerID="e9406d9dc7f85a9a274b160d43844a46acf193bd2ed252ccb1428d25316824e2" exitCode=0
Oct 30 01:15:39.536651 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:39.536264    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk" event={"ID":"3a94e905-9533-4db9-b8d2-5c10538e0d0b","Type":"ContainerDied","Data":"e9406d9dc7f85a9a274b160d43844a46acf193bd2ed252ccb1428d25316824e2"}
Oct 30 01:15:39.536735 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:39.536712    4864 scope.go:122] "RemoveContainer" containerID="e9406d9dc7f85a9a274b160d43844a46acf193bd2ed252ccb1428d25316824e2"
Oct 30 01:15:39.538346 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:39.538317    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:15:39.538411 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:39.538388    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:15:39.538446 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:39.538418    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:15:40.545721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:40.545684    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk" event={"ID":"3a94e905-9533-4db9-b8d2-5c10538e0d0b","Type":"ContainerStarted","Data":"ed07846e19d4ade2537dd402e5b7fe5932e72263c716f8f2036101b74d8854f0"}
Oct 30 01:15:40.731046 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:40.731010    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:15:40.731254 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:40.731229    4864 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 01:15:40.731309 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:40.731292    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk" podUID="3a94e905-9533-4db9-b8d2-5c10538e0d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 01:15:41.731968 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:41.731933    4864 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 01:15:41.732410 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:41.731988    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk" podUID="3a94e905-9533-4db9-b8d2-5c10538e0d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 01:15:42.735093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:42.734974    4864 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 01:15:42.735093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:42.735023    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk" podUID="3a94e905-9533-4db9-b8d2-5c10538e0d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 01:15:43.538537 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:43.538502    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl"
Oct 30 01:15:43.733830 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:43.733795    4864 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 01:15:43.747974 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:43.747931    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk" podUID="3a94e905-9533-4db9-b8d2-5c10538e0d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 01:15:44.731955 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:44.731922    4864 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 01:15:44.732146 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:44.731975    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk" podUID="3a94e905-9533-4db9-b8d2-5c10538e0d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 01:15:45.851458 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:45.851429    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:15:45.852667 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:45.852639    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:15:45.853515 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:15:45.853489    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:15:49.042999 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:15:49.042959    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:16:00.920414 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:00.920373    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"]
Oct 30 01:16:00.924320 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:00.924298    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:00.953107 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:00.953055    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"]
Oct 30 01:16:01.023598 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.023563    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-istio-envoy\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.023799 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.023604    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-workload-certs\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.023799 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.023725    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-workload-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.023799 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.023789    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhwzk\" (UniqueName: \"kubernetes.io/projected/c3816261-e881-443f-aa99-889f28c594a5-kube-api-access-xhwzk\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.023930 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.023903    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-credential-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.023975 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.023942    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3816261-e881-443f-aa99-889f28c594a5-istio-token\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.024045 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.024025    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3816261-e881-443f-aa99-889f28c594a5-istio-podinfo\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.024177 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.024150    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-istio-data\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.024305 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.024216    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3816261-e881-443f-aa99-889f28c594a5-istiod-ca-cert\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.125294 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.125256    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-istio-data\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.125496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.125326    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3816261-e881-443f-aa99-889f28c594a5-istiod-ca-cert\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.125496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.125428    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-istio-envoy\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.125496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.125452    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-workload-certs\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.125676 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.125510    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-workload-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.125676 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.125588    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhwzk\" (UniqueName: \"kubernetes.io/projected/c3816261-e881-443f-aa99-889f28c594a5-kube-api-access-xhwzk\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.125676 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.125642    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-credential-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.125676 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.125669    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3816261-e881-443f-aa99-889f28c594a5-istio-token\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.125887 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.125716    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3816261-e881-443f-aa99-889f28c594a5-istio-podinfo\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.125887 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.125754    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-istio-data\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.125887 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.125796    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-workload-certs\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.126061 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.125900    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-workload-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.126189 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.126168    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-credential-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.126263 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.126226    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3816261-e881-443f-aa99-889f28c594a5-istiod-ca-cert\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.127764 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.127743    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-istio-envoy\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.127999 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.127981    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3816261-e881-443f-aa99-889f28c594a5-istio-podinfo\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.146818 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.146790    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3816261-e881-443f-aa99-889f28c594a5-istio-token\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.146967 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.146949    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhwzk\" (UniqueName: \"kubernetes.io/projected/c3816261-e881-443f-aa99-889f28c594a5-kube-api-access-xhwzk\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.238910 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.238879    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:01.553697 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.553670    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"]
Oct 30 01:16:01.554668 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:16:01.554642    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3816261_e881_443f_aa99_889f28c594a5.slice/crio-85a06657526e8e354bf16b8f02420670c74c3b21eadf71e022d633bb5aeab0bc WatchSource:0}: Error finding container 85a06657526e8e354bf16b8f02420670c74c3b21eadf71e022d633bb5aeab0bc: Status 404 returned error can't find the container with id 85a06657526e8e354bf16b8f02420670c74c3b21eadf71e022d633bb5aeab0bc
Oct 30 01:16:01.557268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.557238    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:16:01.557356 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.557299    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:16:01.557356 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.557330    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:16:01.724389 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.724349    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t" event={"ID":"c3816261-e881-443f-aa99-889f28c594a5","Type":"ContainerStarted","Data":"6c3d4acedee66c449d973670c291900fe46eb282609615654f4b7fac8acc02bd"}
Oct 30 01:16:01.724552 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.724405    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t" event={"ID":"c3816261-e881-443f-aa99-889f28c594a5","Type":"ContainerStarted","Data":"85a06657526e8e354bf16b8f02420670c74c3b21eadf71e022d633bb5aeab0bc"}
Oct 30 01:16:01.766108 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:01.766011    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t" podStartSLOduration=1.7659926989999999 podStartE2EDuration="1.765992699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:01.765060162 +0000 UTC m=+2074.103162073" watchObservedRunningTime="2026-10-30 01:16:01.765992699 +0000 UTC m=+2074.104094615"
Oct 30 01:16:02.244016 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:02.240869    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:02.244016 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:02.240996    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 01:16:02.244016 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:02.241042    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t" podUID="c3816261-e881-443f-aa99-889f28c594a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 01:16:03.239354 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:03.239317    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 01:16:03.239584 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:03.239422    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t" podUID="c3816261-e881-443f-aa99-889f28c594a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 01:16:04.240020 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:04.239983    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 01:16:04.240585 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:04.240043    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t" podUID="c3816261-e881-443f-aa99-889f28c594a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 01:16:05.239233 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:05.239198    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 01:16:05.239501 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:05.239252    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t" podUID="c3816261-e881-443f-aa99-889f28c594a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 01:16:06.240146 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:06.240116    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 01:16:06.240544 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:06.240174    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t" podUID="c3816261-e881-443f-aa99-889f28c594a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 01:16:07.240009 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:07.239979    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 01:16:07.240203 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:07.240027    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t" podUID="c3816261-e881-443f-aa99-889f28c594a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 01:16:08.240198 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:08.240065    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 01:16:08.240686 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:08.240247    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t" podUID="c3816261-e881-443f-aa99-889f28c594a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 01:16:09.244809 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:09.244781    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:09.245280 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:09.245265    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:09.246004 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:09.245987    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:16:19.040016 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:16:19.039978    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:16:28.086579 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:28.086546    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:16:28.101176 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:28.101149    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:16:49.043698 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:16:49.043615    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:16:51.594981 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.594945    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"]
Oct 30 01:16:51.600602 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.600580    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:16:51.617276 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.617247    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"]
Oct 30 01:16:51.623692 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.623666    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-c666c2\"/\"openshift-service-ca.crt\""
Oct 30 01:16:51.623846 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.623707    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-c666c2\"/\"auth-test-model-c666c2-kserve-self-signed-certs\""
Oct 30 01:16:51.624338 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.624320    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-c666c2\"/\"default-dockercfg-ckgqh\""
Oct 30 01:16:51.624338 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.624330    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-c666c2\"/\"kube-root-ca.crt\""
Oct 30 01:16:51.684311 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.684272    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq946\" (UniqueName: \"kubernetes.io/projected/ed407c49-7ffe-47bc-a5aa-e4e336f92130-kube-api-access-wq946\") pod \"auth-test-model-c666c2-kserve-78cd5cf76f-2292r\" (UID: \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:16:51.684494 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.684473    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-tmp-dir\") pod \"auth-test-model-c666c2-kserve-78cd5cf76f-2292r\" (UID: \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:16:51.684553 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.684539    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed407c49-7ffe-47bc-a5aa-e4e336f92130-tls-certs\") pod \"auth-test-model-c666c2-kserve-78cd5cf76f-2292r\" (UID: \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:16:51.684636 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.684611    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-home\") pod \"auth-test-model-c666c2-kserve-78cd5cf76f-2292r\" (UID: \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:16:51.684672 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.684652    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-model-cache\") pod \"auth-test-model-c666c2-kserve-78cd5cf76f-2292r\" (UID: \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:16:51.684706 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.684674    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-dshm\") pod \"auth-test-model-c666c2-kserve-78cd5cf76f-2292r\" (UID: \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:16:51.786670 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.786632    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-tmp-dir\") pod \"auth-test-model-c666c2-kserve-78cd5cf76f-2292r\" (UID: \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:16:51.786842 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.786737    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed407c49-7ffe-47bc-a5aa-e4e336f92130-tls-certs\") pod \"auth-test-model-c666c2-kserve-78cd5cf76f-2292r\" (UID: \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:16:51.786842 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.786815    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-home\") pod \"auth-test-model-c666c2-kserve-78cd5cf76f-2292r\" (UID: \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:16:51.786943 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.786849    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-model-cache\") pod \"auth-test-model-c666c2-kserve-78cd5cf76f-2292r\" (UID: \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:16:51.786943 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.786882    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-dshm\") pod \"auth-test-model-c666c2-kserve-78cd5cf76f-2292r\" (UID: \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:16:51.787052 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.786955    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq946\" (UniqueName: \"kubernetes.io/projected/ed407c49-7ffe-47bc-a5aa-e4e336f92130-kube-api-access-wq946\") pod \"auth-test-model-c666c2-kserve-78cd5cf76f-2292r\" (UID: \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:16:51.787142 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.787119    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-tmp-dir\") pod \"auth-test-model-c666c2-kserve-78cd5cf76f-2292r\" (UID: \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:16:51.787232 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.787199    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-home\") pod \"auth-test-model-c666c2-kserve-78cd5cf76f-2292r\" (UID: \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:16:51.787296 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.787273    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-model-cache\") pod \"auth-test-model-c666c2-kserve-78cd5cf76f-2292r\" (UID: \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:16:51.789253 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.789227    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-dshm\") pod \"auth-test-model-c666c2-kserve-78cd5cf76f-2292r\" (UID: \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:16:51.789853 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.789835    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed407c49-7ffe-47bc-a5aa-e4e336f92130-tls-certs\") pod \"auth-test-model-c666c2-kserve-78cd5cf76f-2292r\" (UID: \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:16:51.810100 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.810044    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq946\" (UniqueName: \"kubernetes.io/projected/ed407c49-7ffe-47bc-a5aa-e4e336f92130-kube-api-access-wq946\") pod \"auth-test-model-c666c2-kserve-78cd5cf76f-2292r\" (UID: \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:16:51.926257 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:51.925877    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:16:52.262793 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:52.262763    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"]
Oct 30 01:16:52.264156 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:16:52.264128    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded407c49_7ffe_47bc_a5aa_e4e336f92130.slice/crio-30c69be82ad63030037cfafc507c50a168b7e13ced83b341841bbffc243e744e WatchSource:0}: Error finding container 30c69be82ad63030037cfafc507c50a168b7e13ced83b341841bbffc243e744e: Status 404 returned error can't find the container with id 30c69be82ad63030037cfafc507c50a168b7e13ced83b341841bbffc243e744e
Oct 30 01:16:53.133177 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:53.133124    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r" event={"ID":"ed407c49-7ffe-47bc-a5aa-e4e336f92130","Type":"ContainerStarted","Data":"87f3e2d11ad054972be57e634fdba51f9c85275a0eb062764ada1044569c3ead"}
Oct 30 01:16:53.133177 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:53.133178    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r" event={"ID":"ed407c49-7ffe-47bc-a5aa-e4e336f92130","Type":"ContainerStarted","Data":"30c69be82ad63030037cfafc507c50a168b7e13ced83b341841bbffc243e744e"}
Oct 30 01:16:53.175130 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:53.175053    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r" podStartSLOduration=2.175038101 podStartE2EDuration="2.175038101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:53.172506397 +0000 UTC m=+2125.510608307" watchObservedRunningTime="2026-10-30 01:16:53.175038101 +0000 UTC m=+2125.513140011"
Oct 30 01:16:53.926980 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:16:53.926942    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:17:01.927326 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:01.927290    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:17:01.941189 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:01.941159    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:17:02.203317 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:02.203291    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:17:08.352642 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.352602    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"]
Oct 30 01:17:08.357635 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.357598    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:08.381120 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.381061    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-bbb402\"/\"openshift-service-ca.crt\""
Oct 30 01:17:08.381282 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.381246    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-bbb402\"/\"kube-root-ca.crt\""
Oct 30 01:17:08.381356 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.381333    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-bbb402\"/\"default-dockercfg-mzbgw\""
Oct 30 01:17:08.381588 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.381566    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-bbb402\"/\"explicit-model-bbb402-kserve-self-signed-certs\""
Oct 30 01:17:08.385117 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.385095    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"]
Oct 30 01:17:08.476754 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.476711    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-home\") pod \"explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg\" (UID: \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\") " pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:08.476940 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.476763    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46081ab-3d02-4f6f-97e9-20daa43d5de3-tls-certs\") pod \"explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg\" (UID: \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\") " pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:08.476940 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.476889    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-dshm\") pod \"explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg\" (UID: \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\") " pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:08.477061 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.477038    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-model-cache\") pod \"explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg\" (UID: \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\") " pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:08.477176 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.477108    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-tmp-dir\") pod \"explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg\" (UID: \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\") " pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:08.477233 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.477217    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65mt2\" (UniqueName: \"kubernetes.io/projected/a46081ab-3d02-4f6f-97e9-20daa43d5de3-kube-api-access-65mt2\") pod \"explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg\" (UID: \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\") " pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:08.579304 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.579263    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-model-cache\") pod \"explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg\" (UID: \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\") " pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:08.579304 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.579316    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-tmp-dir\") pod \"explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg\" (UID: \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\") " pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:08.579614 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.579391    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65mt2\" (UniqueName: \"kubernetes.io/projected/a46081ab-3d02-4f6f-97e9-20daa43d5de3-kube-api-access-65mt2\") pod \"explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg\" (UID: \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\") " pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:08.579614 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.579516    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-home\") pod \"explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg\" (UID: \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\") " pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:08.579614 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.579552    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46081ab-3d02-4f6f-97e9-20daa43d5de3-tls-certs\") pod \"explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg\" (UID: \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\") " pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:08.579771 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.579626    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-dshm\") pod \"explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg\" (UID: \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\") " pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:08.580512 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.580482    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-model-cache\") pod \"explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg\" (UID: \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\") " pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:08.580738 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.580676    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-tmp-dir\") pod \"explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg\" (UID: \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\") " pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:08.580738 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.580731    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-home\") pod \"explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg\" (UID: \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\") " pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:08.583010 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.582981    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-dshm\") pod \"explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg\" (UID: \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\") " pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:08.583562 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.583542    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46081ab-3d02-4f6f-97e9-20daa43d5de3-tls-certs\") pod \"explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg\" (UID: \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\") " pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:08.613064 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.612926    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65mt2\" (UniqueName: \"kubernetes.io/projected/a46081ab-3d02-4f6f-97e9-20daa43d5de3-kube-api-access-65mt2\") pod \"explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg\" (UID: \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\") " pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:08.675094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:08.674156    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:09.010417 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:09.010385    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"]
Oct 30 01:17:09.012626 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:17:09.012594    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda46081ab_3d02_4f6f_97e9_20daa43d5de3.slice/crio-8852868a9615eed4671f6c288b28b663d9264f26a6dc61c01274d223b5377fe1 WatchSource:0}: Error finding container 8852868a9615eed4671f6c288b28b663d9264f26a6dc61c01274d223b5377fe1: Status 404 returned error can't find the container with id 8852868a9615eed4671f6c288b28b663d9264f26a6dc61c01274d223b5377fe1
Oct 30 01:17:09.242280 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:09.242242    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg" event={"ID":"a46081ab-3d02-4f6f-97e9-20daa43d5de3","Type":"ContainerStarted","Data":"940c10e1435f4f5f400ddfbe8355899c2a3decdb2bb63eb38c2c34797be7ad08"}
Oct 30 01:17:09.242491 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:09.242292    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg" event={"ID":"a46081ab-3d02-4f6f-97e9-20daa43d5de3","Type":"ContainerStarted","Data":"8852868a9615eed4671f6c288b28b663d9264f26a6dc61c01274d223b5377fe1"}
Oct 30 01:17:09.280827 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:09.280702    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg" podStartSLOduration=1.280679105 podStartE2EDuration="1.280679105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:09.27946451 +0000 UTC m=+2141.617566452" watchObservedRunningTime="2026-10-30 01:17:09.280679105 +0000 UTC m=+2141.618781016"
Oct 30 01:17:09.675330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:09.675238    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:18.675391 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:18.675352    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:18.688036 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:18.688009    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:19.043629 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:17:19.043593    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:17:19.320885 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:19.320810    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:23.092116 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:23.087048    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"]
Oct 30 01:17:38.110185 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.110125    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg" podUID="a46081ab-3d02-4f6f-97e9-20daa43d5de3" containerName="main" containerID="cri-o://940c10e1435f4f5f400ddfbe8355899c2a3decdb2bb63eb38c2c34797be7ad08" gracePeriod=45
Oct 30 01:17:38.398086 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.398044    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:38.483093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.482831    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-dshm\") pod \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\" (UID: \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\") "
Oct 30 01:17:38.483302 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.483117    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-model-cache\") pod \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\" (UID: \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\") "
Oct 30 01:17:38.483372 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.483339    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a46081ab-3d02-4f6f-97e9-20daa43d5de3-tls-certs\" (UniqueName: \"kubernetes.io/secret/a46081ab-3d02-4f6f-97e9-20daa43d5de3-tls-certs\") pod \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\" (UID: \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\") "
Oct 30 01:17:38.487503 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.483517    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-home\" (UniqueName: \"kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-home\") pod \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\" (UID: \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\") "
Oct 30 01:17:38.487503 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.483577    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a46081ab-3d02-4f6f-97e9-20daa43d5de3-kube-api-access-65mt2\" (UniqueName: \"kubernetes.io/projected/a46081ab-3d02-4f6f-97e9-20daa43d5de3-kube-api-access-65mt2\") pod \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\" (UID: \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\") "
Oct 30 01:17:38.487503 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.483695    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-tmp-dir\") pod \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\" (UID: \"a46081ab-3d02-4f6f-97e9-20daa43d5de3\") "
Oct 30 01:17:38.487503 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.484562    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-home" pod "a46081ab-3d02-4f6f-97e9-20daa43d5de3" (UID: "a46081ab-3d02-4f6f-97e9-20daa43d5de3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:38.487503 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.484793    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-model-cache" pod "a46081ab-3d02-4f6f-97e9-20daa43d5de3" (UID: "a46081ab-3d02-4f6f-97e9-20daa43d5de3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:38.487503 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.485149    4864 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-home\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:17:38.487503 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.485189    4864 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-model-cache\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:17:38.487503 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.485238    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-tmp-dir" pod "a46081ab-3d02-4f6f-97e9-20daa43d5de3" (UID: "a46081ab-3d02-4f6f-97e9-20daa43d5de3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:38.490205 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.490170    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a46081ab-3d02-4f6f-97e9-20daa43d5de3-tls-certs" pod "a46081ab-3d02-4f6f-97e9-20daa43d5de3" (UID: "a46081ab-3d02-4f6f-97e9-20daa43d5de3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:38.491772 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.491730    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-dshm" pod "a46081ab-3d02-4f6f-97e9-20daa43d5de3" (UID: "a46081ab-3d02-4f6f-97e9-20daa43d5de3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:38.492922 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.492879    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a46081ab-3d02-4f6f-97e9-20daa43d5de3-kube-api-access-65mt2" pod "a46081ab-3d02-4f6f-97e9-20daa43d5de3" (UID: "a46081ab-3d02-4f6f-97e9-20daa43d5de3"). InnerVolumeSpecName "kube-api-access-65mt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:38.495713 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.495682    4864 generic.go:356] "Generic (PLEG): container finished" podID="a46081ab-3d02-4f6f-97e9-20daa43d5de3" containerID="940c10e1435f4f5f400ddfbe8355899c2a3decdb2bb63eb38c2c34797be7ad08" exitCode=0
Oct 30 01:17:38.495850 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.495779    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg" event={"ID":"a46081ab-3d02-4f6f-97e9-20daa43d5de3","Type":"ContainerDied","Data":"940c10e1435f4f5f400ddfbe8355899c2a3decdb2bb63eb38c2c34797be7ad08"}
Oct 30 01:17:38.495850 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.495817    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg" event={"ID":"a46081ab-3d02-4f6f-97e9-20daa43d5de3","Type":"ContainerDied","Data":"8852868a9615eed4671f6c288b28b663d9264f26a6dc61c01274d223b5377fe1"}
Oct 30 01:17:38.495850 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.495838    4864 scope.go:122] "RemoveContainer" containerID="940c10e1435f4f5f400ddfbe8355899c2a3decdb2bb63eb38c2c34797be7ad08"
Oct 30 01:17:38.496219 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.496200    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"
Oct 30 01:17:38.525246 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.525222    4864 scope.go:122] "RemoveContainer" containerID="940c10e1435f4f5f400ddfbe8355899c2a3decdb2bb63eb38c2c34797be7ad08"
Oct 30 01:17:38.525546 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:17:38.525519    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"940c10e1435f4f5f400ddfbe8355899c2a3decdb2bb63eb38c2c34797be7ad08\": container with ID starting with 940c10e1435f4f5f400ddfbe8355899c2a3decdb2bb63eb38c2c34797be7ad08 not found: ID does not exist" containerID="940c10e1435f4f5f400ddfbe8355899c2a3decdb2bb63eb38c2c34797be7ad08"
Oct 30 01:17:38.525651 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.525553    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940c10e1435f4f5f400ddfbe8355899c2a3decdb2bb63eb38c2c34797be7ad08"} err="failed to get container status \"940c10e1435f4f5f400ddfbe8355899c2a3decdb2bb63eb38c2c34797be7ad08\": rpc error: code = NotFound desc = could not find container \"940c10e1435f4f5f400ddfbe8355899c2a3decdb2bb63eb38c2c34797be7ad08\": container with ID starting with 940c10e1435f4f5f400ddfbe8355899c2a3decdb2bb63eb38c2c34797be7ad08 not found: ID does not exist"
Oct 30 01:17:38.554663 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.554624    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"]
Oct 30 01:17:38.559263 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.559223    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg"]
Oct 30 01:17:38.586949 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.586905    4864 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46081ab-3d02-4f6f-97e9-20daa43d5de3-tls-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:17:38.586949 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.586941    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65mt2\" (UniqueName: \"kubernetes.io/projected/a46081ab-3d02-4f6f-97e9-20daa43d5de3-kube-api-access-65mt2\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:17:38.587210 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.586980    4864 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-tmp-dir\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:17:38.587210 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:38.586996    4864 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46081ab-3d02-4f6f-97e9-20daa43d5de3-dshm\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:17:39.310248 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:39.310204    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-bbb402/explicit-model-bbb402-kserve-cbb7dc8bc-mrqmg" podUID="a46081ab-3d02-4f6f-97e9-20daa43d5de3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:17:39.793830 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:17:39.793796    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a46081ab-3d02-4f6f-97e9-20daa43d5de3" path="/var/lib/kubelet/pods/a46081ab-3d02-4f6f-97e9-20daa43d5de3/volumes"
Oct 30 01:17:49.041980 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:17:49.041942    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:18:02.785440 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:02.785402    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"]
Oct 30 01:18:02.786458 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:02.786435    4864 cpu_manager.go:414] "containerMap: removing container" podUID="a46081ab-3d02-4f6f-97e9-20daa43d5de3" containerName="main"
Oct 30 01:18:02.786522 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:02.786460    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a46081ab-3d02-4f6f-97e9-20daa43d5de3" containerName="main"
Oct 30 01:18:02.786639 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:02.786621    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="a46081ab-3d02-4f6f-97e9-20daa43d5de3" containerName="main"
Oct 30 01:18:02.790585 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:02.790563    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:02.809595 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:02.809550    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"]
Oct 30 01:18:02.814694 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:02.814671    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-6d45f8\"/\"kube-root-ca.crt\""
Oct 30 01:18:02.814961 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:02.814932    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-6d45f8\"/\"default-dockercfg-nqffj\""
Oct 30 01:18:02.898920 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:02.898884    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-6d45f8\"/\"noten-model-6d45f8-kserve-self-signed-certs\""
Oct 30 01:18:02.901621 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:02.901597    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p89s\" (UniqueName: \"kubernetes.io/projected/0dbf1a8d-c934-43fa-8a04-812019a09865-kube-api-access-6p89s\") pod \"noten-model-6d45f8-kserve-7d86b89d45-kspss\" (UID: \"0dbf1a8d-c934-43fa-8a04-812019a09865\") " pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:02.901696 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:02.901646    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-dshm\") pod \"noten-model-6d45f8-kserve-7d86b89d45-kspss\" (UID: \"0dbf1a8d-c934-43fa-8a04-812019a09865\") " pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:02.901743 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:02.901731    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-tmp-dir\") pod \"noten-model-6d45f8-kserve-7d86b89d45-kspss\" (UID: \"0dbf1a8d-c934-43fa-8a04-812019a09865\") " pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:02.901779 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:02.901760    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dbf1a8d-c934-43fa-8a04-812019a09865-tls-certs\") pod \"noten-model-6d45f8-kserve-7d86b89d45-kspss\" (UID: \"0dbf1a8d-c934-43fa-8a04-812019a09865\") " pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:02.901822 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:02.901802    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-home\") pod \"noten-model-6d45f8-kserve-7d86b89d45-kspss\" (UID: \"0dbf1a8d-c934-43fa-8a04-812019a09865\") " pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:02.901919 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:02.901905    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-model-cache\") pod \"noten-model-6d45f8-kserve-7d86b89d45-kspss\" (UID: \"0dbf1a8d-c934-43fa-8a04-812019a09865\") " pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:02.904799 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:02.904779    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-6d45f8\"/\"openshift-service-ca.crt\""
Oct 30 01:18:03.003007 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:03.002966    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-tmp-dir\") pod \"noten-model-6d45f8-kserve-7d86b89d45-kspss\" (UID: \"0dbf1a8d-c934-43fa-8a04-812019a09865\") " pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:03.003213 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:03.003010    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dbf1a8d-c934-43fa-8a04-812019a09865-tls-certs\") pod \"noten-model-6d45f8-kserve-7d86b89d45-kspss\" (UID: \"0dbf1a8d-c934-43fa-8a04-812019a09865\") " pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:03.003213 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:03.003092    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-home\") pod \"noten-model-6d45f8-kserve-7d86b89d45-kspss\" (UID: \"0dbf1a8d-c934-43fa-8a04-812019a09865\") " pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:03.003213 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:03.003161    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-model-cache\") pod \"noten-model-6d45f8-kserve-7d86b89d45-kspss\" (UID: \"0dbf1a8d-c934-43fa-8a04-812019a09865\") " pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:03.003383 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:03.003358    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-tmp-dir\") pod \"noten-model-6d45f8-kserve-7d86b89d45-kspss\" (UID: \"0dbf1a8d-c934-43fa-8a04-812019a09865\") " pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:03.003437 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:03.003374    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p89s\" (UniqueName: \"kubernetes.io/projected/0dbf1a8d-c934-43fa-8a04-812019a09865-kube-api-access-6p89s\") pod \"noten-model-6d45f8-kserve-7d86b89d45-kspss\" (UID: \"0dbf1a8d-c934-43fa-8a04-812019a09865\") " pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:03.003493 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:03.003453    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-home\") pod \"noten-model-6d45f8-kserve-7d86b89d45-kspss\" (UID: \"0dbf1a8d-c934-43fa-8a04-812019a09865\") " pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:03.003546 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:03.003500    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-model-cache\") pod \"noten-model-6d45f8-kserve-7d86b89d45-kspss\" (UID: \"0dbf1a8d-c934-43fa-8a04-812019a09865\") " pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:03.003598 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:03.003547    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-dshm\") pod \"noten-model-6d45f8-kserve-7d86b89d45-kspss\" (UID: \"0dbf1a8d-c934-43fa-8a04-812019a09865\") " pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:03.005755 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:03.005732    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dbf1a8d-c934-43fa-8a04-812019a09865-tls-certs\") pod \"noten-model-6d45f8-kserve-7d86b89d45-kspss\" (UID: \"0dbf1a8d-c934-43fa-8a04-812019a09865\") " pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:03.005890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:03.005851    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-dshm\") pod \"noten-model-6d45f8-kserve-7d86b89d45-kspss\" (UID: \"0dbf1a8d-c934-43fa-8a04-812019a09865\") " pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:03.024745 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:03.024719    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p89s\" (UniqueName: \"kubernetes.io/projected/0dbf1a8d-c934-43fa-8a04-812019a09865-kube-api-access-6p89s\") pod \"noten-model-6d45f8-kserve-7d86b89d45-kspss\" (UID: \"0dbf1a8d-c934-43fa-8a04-812019a09865\") " pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:03.118197 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:03.117345    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:03.467350 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:03.467320    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"]
Oct 30 01:18:03.468421 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:18:03.468386    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0dbf1a8d_c934_43fa_8a04_812019a09865.slice/crio-7ebb1417daccd259b7dfa7abb6238896b7af87b3e57146a5af1931b2b4079d85 WatchSource:0}: Error finding container 7ebb1417daccd259b7dfa7abb6238896b7af87b3e57146a5af1931b2b4079d85: Status 404 returned error can't find the container with id 7ebb1417daccd259b7dfa7abb6238896b7af87b3e57146a5af1931b2b4079d85
Oct 30 01:18:03.769567 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:03.769476    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss" event={"ID":"0dbf1a8d-c934-43fa-8a04-812019a09865","Type":"ContainerStarted","Data":"488a3f585eab0fbec76b6ec760c26bdd14fc84cbb5eb8838ad0f154860393d6e"}
Oct 30 01:18:03.769567 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:03.769523    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss" event={"ID":"0dbf1a8d-c934-43fa-8a04-812019a09865","Type":"ContainerStarted","Data":"7ebb1417daccd259b7dfa7abb6238896b7af87b3e57146a5af1931b2b4079d85"}
Oct 30 01:18:03.809397 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:03.809343    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss" podStartSLOduration=1.809329272 podStartE2EDuration="1.809329272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:03.806817723 +0000 UTC m=+2196.144919633" watchObservedRunningTime="2026-10-30 01:18:03.809329272 +0000 UTC m=+2196.147431181"
Oct 30 01:18:04.123515 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:04.123438    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:13.123875 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:13.123755    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:13.141358 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:13.141329    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:13.850765 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:13.850734    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:19.038866 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:18:19.038829    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:18:31.080146 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:31.080113    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"]
Oct 30 01:18:46.101178 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:46.101143    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss" podUID="0dbf1a8d-c934-43fa-8a04-812019a09865" containerName="main" containerID="cri-o://488a3f585eab0fbec76b6ec760c26bdd14fc84cbb5eb8838ad0f154860393d6e" gracePeriod=45
Oct 30 01:18:46.388143 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:46.388118    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:46.489481 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:46.489448    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-home\" (UniqueName: \"kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-home\") pod \"0dbf1a8d-c934-43fa-8a04-812019a09865\" (UID: \"0dbf1a8d-c934-43fa-8a04-812019a09865\") "
Oct 30 01:18:46.489646 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:46.489508    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dbf1a8d-c934-43fa-8a04-812019a09865-tls-certs\" (UniqueName: \"kubernetes.io/secret/0dbf1a8d-c934-43fa-8a04-812019a09865-tls-certs\") pod \"0dbf1a8d-c934-43fa-8a04-812019a09865\" (UID: \"0dbf1a8d-c934-43fa-8a04-812019a09865\") "
Oct 30 01:18:46.489646 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:46.489571    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dbf1a8d-c934-43fa-8a04-812019a09865-kube-api-access-6p89s\" (UniqueName: \"kubernetes.io/projected/0dbf1a8d-c934-43fa-8a04-812019a09865-kube-api-access-6p89s\") pod \"0dbf1a8d-c934-43fa-8a04-812019a09865\" (UID: \"0dbf1a8d-c934-43fa-8a04-812019a09865\") "
Oct 30 01:18:46.489721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:46.489693    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-tmp-dir\") pod \"0dbf1a8d-c934-43fa-8a04-812019a09865\" (UID: \"0dbf1a8d-c934-43fa-8a04-812019a09865\") "
Oct 30 01:18:46.489755 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:46.489732    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-dshm\") pod \"0dbf1a8d-c934-43fa-8a04-812019a09865\" (UID: \"0dbf1a8d-c934-43fa-8a04-812019a09865\") "
Oct 30 01:18:46.489797 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:46.489754    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-home" pod "0dbf1a8d-c934-43fa-8a04-812019a09865" (UID: "0dbf1a8d-c934-43fa-8a04-812019a09865"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:46.489875 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:46.489861    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-model-cache\") pod \"0dbf1a8d-c934-43fa-8a04-812019a09865\" (UID: \"0dbf1a8d-c934-43fa-8a04-812019a09865\") "
Oct 30 01:18:46.489975 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:46.489931    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-tmp-dir" pod "0dbf1a8d-c934-43fa-8a04-812019a09865" (UID: "0dbf1a8d-c934-43fa-8a04-812019a09865"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:46.490195 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:46.490162    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-model-cache" pod "0dbf1a8d-c934-43fa-8a04-812019a09865" (UID: "0dbf1a8d-c934-43fa-8a04-812019a09865"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:46.490380 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:46.490363    4864 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-model-cache\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:18:46.490741 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:46.490386    4864 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-home\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:18:46.490741 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:46.490399    4864 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-tmp-dir\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:18:46.491921 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:46.491891    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-dshm" pod "0dbf1a8d-c934-43fa-8a04-812019a09865" (UID: "0dbf1a8d-c934-43fa-8a04-812019a09865"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:46.491921 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:46.491910    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dbf1a8d-c934-43fa-8a04-812019a09865-tls-certs" pod "0dbf1a8d-c934-43fa-8a04-812019a09865" (UID: "0dbf1a8d-c934-43fa-8a04-812019a09865"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:18:46.492102 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:46.491915    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dbf1a8d-c934-43fa-8a04-812019a09865-kube-api-access-6p89s" pod "0dbf1a8d-c934-43fa-8a04-812019a09865" (UID: "0dbf1a8d-c934-43fa-8a04-812019a09865"). InnerVolumeSpecName "kube-api-access-6p89s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:46.591143 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:46.591108    4864 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dbf1a8d-c934-43fa-8a04-812019a09865-tls-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:18:46.591143 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:46.591138    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p89s\" (UniqueName: \"kubernetes.io/projected/0dbf1a8d-c934-43fa-8a04-812019a09865-kube-api-access-6p89s\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:18:46.591143 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:46.591148    4864 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dbf1a8d-c934-43fa-8a04-812019a09865-dshm\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:18:47.104143 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:47.104105    4864 generic.go:356] "Generic (PLEG): container finished" podID="0dbf1a8d-c934-43fa-8a04-812019a09865" containerID="488a3f585eab0fbec76b6ec760c26bdd14fc84cbb5eb8838ad0f154860393d6e" exitCode=0
Oct 30 01:18:47.104646 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:47.104152    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"
Oct 30 01:18:47.104646 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:47.104185    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss" event={"ID":"0dbf1a8d-c934-43fa-8a04-812019a09865","Type":"ContainerDied","Data":"488a3f585eab0fbec76b6ec760c26bdd14fc84cbb5eb8838ad0f154860393d6e"}
Oct 30 01:18:47.104646 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:47.104241    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss" event={"ID":"0dbf1a8d-c934-43fa-8a04-812019a09865","Type":"ContainerDied","Data":"7ebb1417daccd259b7dfa7abb6238896b7af87b3e57146a5af1931b2b4079d85"}
Oct 30 01:18:47.104646 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:47.104268    4864 scope.go:122] "RemoveContainer" containerID="488a3f585eab0fbec76b6ec760c26bdd14fc84cbb5eb8838ad0f154860393d6e"
Oct 30 01:18:47.114824 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:47.114806    4864 scope.go:122] "RemoveContainer" containerID="488a3f585eab0fbec76b6ec760c26bdd14fc84cbb5eb8838ad0f154860393d6e"
Oct 30 01:18:47.115150 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:18:47.115128    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"488a3f585eab0fbec76b6ec760c26bdd14fc84cbb5eb8838ad0f154860393d6e\": container with ID starting with 488a3f585eab0fbec76b6ec760c26bdd14fc84cbb5eb8838ad0f154860393d6e not found: ID does not exist" containerID="488a3f585eab0fbec76b6ec760c26bdd14fc84cbb5eb8838ad0f154860393d6e"
Oct 30 01:18:47.115246 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:47.115155    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"488a3f585eab0fbec76b6ec760c26bdd14fc84cbb5eb8838ad0f154860393d6e"} err="failed to get container status \"488a3f585eab0fbec76b6ec760c26bdd14fc84cbb5eb8838ad0f154860393d6e\": rpc error: code = NotFound desc = could not find container \"488a3f585eab0fbec76b6ec760c26bdd14fc84cbb5eb8838ad0f154860393d6e\": container with ID starting with 488a3f585eab0fbec76b6ec760c26bdd14fc84cbb5eb8838ad0f154860393d6e not found: ID does not exist"
Oct 30 01:18:47.151793 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:47.151759    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"]
Oct 30 01:18:47.161648 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:47.161622    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-6d45f8/noten-model-6d45f8-kserve-7d86b89d45-kspss"]
Oct 30 01:18:47.793382 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:18:47.793351    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dbf1a8d-c934-43fa-8a04-812019a09865" path="/var/lib/kubelet/pods/0dbf1a8d-c934-43fa-8a04-812019a09865/volumes"
Oct 30 01:18:49.042251 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:18:49.042213    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:19:19.044975 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:19:19.044936    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:19:40.617287 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:40.617201    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"]
Oct 30 01:19:40.617659 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:40.617422    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg" podUID="dc29530c-080c-4698-a2f3-7300a77e9804" containerName="istio-proxy" containerID="cri-o://21b2a38102167dbe0dd4c00c6bcdf6dc5fa3a98a1c52a1f71ba10a9ec6f74e2d" gracePeriod=30
Oct 30 01:19:45.915544 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:45.915516    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:19:46.059011 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.058972    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc29530c-080c-4698-a2f3-7300a77e9804-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc29530c-080c-4698-a2f3-7300a77e9804-istio-podinfo\") pod \"dc29530c-080c-4698-a2f3-7300a77e9804\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") "
Oct 30 01:19:46.059225 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.059021    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-workload-socket\") pod \"dc29530c-080c-4698-a2f3-7300a77e9804\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") "
Oct 30 01:19:46.059225 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.059066    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-istio-data\") pod \"dc29530c-080c-4698-a2f3-7300a77e9804\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") "
Oct 30 01:19:46.059225 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.059127    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc29530c-080c-4698-a2f3-7300a77e9804-kube-api-access-926d8\" (UniqueName: \"kubernetes.io/projected/dc29530c-080c-4698-a2f3-7300a77e9804-kube-api-access-926d8\") pod \"dc29530c-080c-4698-a2f3-7300a77e9804\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") "
Oct 30 01:19:46.059225 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.059163    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-istio-envoy\") pod \"dc29530c-080c-4698-a2f3-7300a77e9804\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") "
Oct 30 01:19:46.059447 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.059240    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-credential-socket\") pod \"dc29530c-080c-4698-a2f3-7300a77e9804\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") "
Oct 30 01:19:46.059447 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.059313    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc29530c-080c-4698-a2f3-7300a77e9804-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc29530c-080c-4698-a2f3-7300a77e9804-istiod-ca-cert\") pod \"dc29530c-080c-4698-a2f3-7300a77e9804\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") "
Oct 30 01:19:46.059447 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.059394    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-workload-certs\") pod \"dc29530c-080c-4698-a2f3-7300a77e9804\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") "
Oct 30 01:19:46.059447 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.059397    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-istio-data" pod "dc29530c-080c-4698-a2f3-7300a77e9804" (UID: "dc29530c-080c-4698-a2f3-7300a77e9804"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:46.059640 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.059504    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc29530c-080c-4698-a2f3-7300a77e9804-istio-token\" (UniqueName: \"kubernetes.io/projected/dc29530c-080c-4698-a2f3-7300a77e9804-istio-token\") pod \"dc29530c-080c-4698-a2f3-7300a77e9804\" (UID: \"dc29530c-080c-4698-a2f3-7300a77e9804\") "
Oct 30 01:19:46.059730 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.059692    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-workload-certs" pod "dc29530c-080c-4698-a2f3-7300a77e9804" (UID: "dc29530c-080c-4698-a2f3-7300a77e9804"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:46.059785 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.059736    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc29530c-080c-4698-a2f3-7300a77e9804-istiod-ca-cert" pod "dc29530c-080c-4698-a2f3-7300a77e9804" (UID: "dc29530c-080c-4698-a2f3-7300a77e9804"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:46.060219 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.060134    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc29530c-080c-4698-a2f3-7300a77e9804-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:46.060219 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.060159    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:46.060219 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.060174    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:46.060469 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.060249    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-credential-socket" pod "dc29530c-080c-4698-a2f3-7300a77e9804" (UID: "dc29530c-080c-4698-a2f3-7300a77e9804"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:46.060469 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.060370    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-workload-socket" pod "dc29530c-080c-4698-a2f3-7300a77e9804" (UID: "dc29530c-080c-4698-a2f3-7300a77e9804"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:46.061462 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.061441    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc29530c-080c-4698-a2f3-7300a77e9804-istio-podinfo" pod "dc29530c-080c-4698-a2f3-7300a77e9804" (UID: "dc29530c-080c-4698-a2f3-7300a77e9804"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:19:46.061788 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.061762    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc29530c-080c-4698-a2f3-7300a77e9804-istio-token" pod "dc29530c-080c-4698-a2f3-7300a77e9804" (UID: "dc29530c-080c-4698-a2f3-7300a77e9804"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:46.061906 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.061833    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-istio-envoy" pod "dc29530c-080c-4698-a2f3-7300a77e9804" (UID: "dc29530c-080c-4698-a2f3-7300a77e9804"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:46.061996 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.061974    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc29530c-080c-4698-a2f3-7300a77e9804-kube-api-access-926d8" pod "dc29530c-080c-4698-a2f3-7300a77e9804" (UID: "dc29530c-080c-4698-a2f3-7300a77e9804"). InnerVolumeSpecName "kube-api-access-926d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:46.161574 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.161496    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc29530c-080c-4698-a2f3-7300a77e9804-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:46.161574 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.161526    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:46.161574 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.161537    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-926d8\" (UniqueName: \"kubernetes.io/projected/dc29530c-080c-4698-a2f3-7300a77e9804-kube-api-access-926d8\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:46.161574 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.161547    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:46.161574 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.161555    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc29530c-080c-4698-a2f3-7300a77e9804-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:46.161574 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.161567    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc29530c-080c-4698-a2f3-7300a77e9804-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:46.531311 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.531278    4864 generic.go:356] "Generic (PLEG): container finished" podID="dc29530c-080c-4698-a2f3-7300a77e9804" containerID="21b2a38102167dbe0dd4c00c6bcdf6dc5fa3a98a1c52a1f71ba10a9ec6f74e2d" exitCode=0
Oct 30 01:19:46.531484 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.531324    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg" event={"ID":"dc29530c-080c-4698-a2f3-7300a77e9804","Type":"ContainerDied","Data":"21b2a38102167dbe0dd4c00c6bcdf6dc5fa3a98a1c52a1f71ba10a9ec6f74e2d"}
Oct 30 01:19:46.531484 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.531332    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"
Oct 30 01:19:46.531484 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.531375    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg" event={"ID":"dc29530c-080c-4698-a2f3-7300a77e9804","Type":"ContainerDied","Data":"f5864a786d716e6091c9ee15af9731950bae7f6685e643d0a5fed967bc050aa7"}
Oct 30 01:19:46.531484 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.531398    4864 scope.go:122] "RemoveContainer" containerID="21b2a38102167dbe0dd4c00c6bcdf6dc5fa3a98a1c52a1f71ba10a9ec6f74e2d"
Oct 30 01:19:46.542457 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.542439    4864 scope.go:122] "RemoveContainer" containerID="21b2a38102167dbe0dd4c00c6bcdf6dc5fa3a98a1c52a1f71ba10a9ec6f74e2d"
Oct 30 01:19:46.542742 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:19:46.542723    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21b2a38102167dbe0dd4c00c6bcdf6dc5fa3a98a1c52a1f71ba10a9ec6f74e2d\": container with ID starting with 21b2a38102167dbe0dd4c00c6bcdf6dc5fa3a98a1c52a1f71ba10a9ec6f74e2d not found: ID does not exist" containerID="21b2a38102167dbe0dd4c00c6bcdf6dc5fa3a98a1c52a1f71ba10a9ec6f74e2d"
Oct 30 01:19:46.542792 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.542753    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b2a38102167dbe0dd4c00c6bcdf6dc5fa3a98a1c52a1f71ba10a9ec6f74e2d"} err="failed to get container status \"21b2a38102167dbe0dd4c00c6bcdf6dc5fa3a98a1c52a1f71ba10a9ec6f74e2d\": rpc error: code = NotFound desc = could not find container \"21b2a38102167dbe0dd4c00c6bcdf6dc5fa3a98a1c52a1f71ba10a9ec6f74e2d\": container with ID starting with 21b2a38102167dbe0dd4c00c6bcdf6dc5fa3a98a1c52a1f71ba10a9ec6f74e2d not found: ID does not exist"
Oct 30 01:19:46.601045 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.601000    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"]
Oct 30 01:19:46.610656 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:46.610624    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lfpdg"]
Oct 30 01:19:47.778443 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:47.778413    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"]
Oct 30 01:19:47.778848 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:47.778657    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq" podUID="b5bec735-88e4-4b14-899d-5e8158d2638c" containerName="istio-proxy" containerID="cri-o://659b6a3e82e056ddf66f729280fa5d8c18442a0b40fe385c211306e4ee04ac32" gracePeriod=30
Oct 30 01:19:47.796856 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:47.796827    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc29530c-080c-4698-a2f3-7300a77e9804" path="/var/lib/kubelet/pods/dc29530c-080c-4698-a2f3-7300a77e9804/volumes"
Oct 30 01:19:49.044137 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:19:49.044102    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:19:53.469578 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.469554    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:19:53.581426 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.581329    4864 generic.go:356] "Generic (PLEG): container finished" podID="b5bec735-88e4-4b14-899d-5e8158d2638c" containerID="659b6a3e82e056ddf66f729280fa5d8c18442a0b40fe385c211306e4ee04ac32" exitCode=0
Oct 30 01:19:53.581426 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.581402    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"
Oct 30 01:19:53.581603 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.581421    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq" event={"ID":"b5bec735-88e4-4b14-899d-5e8158d2638c","Type":"ContainerDied","Data":"659b6a3e82e056ddf66f729280fa5d8c18442a0b40fe385c211306e4ee04ac32"}
Oct 30 01:19:53.581603 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.581483    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq" event={"ID":"b5bec735-88e4-4b14-899d-5e8158d2638c","Type":"ContainerDied","Data":"5fc5aabc0787df693841cb81e1ac362ddc885755a9a28012f704ad0138a064b4"}
Oct 30 01:19:53.581603 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.581512    4864 scope.go:122] "RemoveContainer" containerID="659b6a3e82e056ddf66f729280fa5d8c18442a0b40fe385c211306e4ee04ac32"
Oct 30 01:19:53.589488 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.589467    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-envoy\") pod \"b5bec735-88e4-4b14-899d-5e8158d2638c\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") "
Oct 30 01:19:53.589653 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.589507    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-workload-socket\") pod \"b5bec735-88e4-4b14-899d-5e8158d2638c\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") "
Oct 30 01:19:53.589653 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.589558    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-workload-certs\") pod \"b5bec735-88e4-4b14-899d-5e8158d2638c\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") "
Oct 30 01:19:53.589653 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.589594    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-token\" (UniqueName: \"kubernetes.io/projected/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-token\") pod \"b5bec735-88e4-4b14-899d-5e8158d2638c\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") "
Oct 30 01:19:53.589653 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.589620    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5bec735-88e4-4b14-899d-5e8158d2638c-kube-api-access-99s4b\" (UniqueName: \"kubernetes.io/projected/b5bec735-88e4-4b14-899d-5e8158d2638c-kube-api-access-99s4b\") pod \"b5bec735-88e4-4b14-899d-5e8158d2638c\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") "
Oct 30 01:19:53.589903 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.589700    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5bec735-88e4-4b14-899d-5e8158d2638c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5bec735-88e4-4b14-899d-5e8158d2638c-istiod-ca-cert\") pod \"b5bec735-88e4-4b14-899d-5e8158d2638c\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") "
Oct 30 01:19:53.589903 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.589879    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-workload-socket" pod "b5bec735-88e4-4b14-899d-5e8158d2638c" (UID: "b5bec735-88e4-4b14-899d-5e8158d2638c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:53.590012 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.589926    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-data\") pod \"b5bec735-88e4-4b14-899d-5e8158d2638c\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") "
Oct 30 01:19:53.590012 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.589981    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-podinfo\") pod \"b5bec735-88e4-4b14-899d-5e8158d2638c\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") "
Oct 30 01:19:53.590137 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.590050    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-credential-socket\") pod \"b5bec735-88e4-4b14-899d-5e8158d2638c\" (UID: \"b5bec735-88e4-4b14-899d-5e8158d2638c\") "
Oct 30 01:19:53.590602 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.590374    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-workload-certs" pod "b5bec735-88e4-4b14-899d-5e8158d2638c" (UID: "b5bec735-88e4-4b14-899d-5e8158d2638c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:53.590602 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.590460    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5bec735-88e4-4b14-899d-5e8158d2638c-istiod-ca-cert" pod "b5bec735-88e4-4b14-899d-5e8158d2638c" (UID: "b5bec735-88e4-4b14-899d-5e8158d2638c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:53.590761 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.590677    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-credential-socket" pod "b5bec735-88e4-4b14-899d-5e8158d2638c" (UID: "b5bec735-88e4-4b14-899d-5e8158d2638c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:53.590819 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.590767    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-data" pod "b5bec735-88e4-4b14-899d-5e8158d2638c" (UID: "b5bec735-88e4-4b14-899d-5e8158d2638c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:53.590819 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.590782    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:53.590926 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.590816    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:53.590926 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.590832    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5bec735-88e4-4b14-899d-5e8158d2638c-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:53.595330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.595216    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-podinfo" pod "b5bec735-88e4-4b14-899d-5e8158d2638c" (UID: "b5bec735-88e4-4b14-899d-5e8158d2638c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:19:53.595330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.595227    4864 scope.go:122] "RemoveContainer" containerID="659b6a3e82e056ddf66f729280fa5d8c18442a0b40fe385c211306e4ee04ac32"
Oct 30 01:19:53.595330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.595221    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-envoy" pod "b5bec735-88e4-4b14-899d-5e8158d2638c" (UID: "b5bec735-88e4-4b14-899d-5e8158d2638c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:53.595330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.595245    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-token" pod "b5bec735-88e4-4b14-899d-5e8158d2638c" (UID: "b5bec735-88e4-4b14-899d-5e8158d2638c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:53.595559 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:19:53.595533    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"659b6a3e82e056ddf66f729280fa5d8c18442a0b40fe385c211306e4ee04ac32\": container with ID starting with 659b6a3e82e056ddf66f729280fa5d8c18442a0b40fe385c211306e4ee04ac32 not found: ID does not exist" containerID="659b6a3e82e056ddf66f729280fa5d8c18442a0b40fe385c211306e4ee04ac32"
Oct 30 01:19:53.595609 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.595571    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"659b6a3e82e056ddf66f729280fa5d8c18442a0b40fe385c211306e4ee04ac32"} err="failed to get container status \"659b6a3e82e056ddf66f729280fa5d8c18442a0b40fe385c211306e4ee04ac32\": rpc error: code = NotFound desc = could not find container \"659b6a3e82e056ddf66f729280fa5d8c18442a0b40fe385c211306e4ee04ac32\": container with ID starting with 659b6a3e82e056ddf66f729280fa5d8c18442a0b40fe385c211306e4ee04ac32 not found: ID does not exist"
Oct 30 01:19:53.596157 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.596134    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5bec735-88e4-4b14-899d-5e8158d2638c-kube-api-access-99s4b" pod "b5bec735-88e4-4b14-899d-5e8158d2638c" (UID: "b5bec735-88e4-4b14-899d-5e8158d2638c"). InnerVolumeSpecName "kube-api-access-99s4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:53.692457 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.692418    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:53.692457 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.692457    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:53.692746 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.692472    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99s4b\" (UniqueName: \"kubernetes.io/projected/b5bec735-88e4-4b14-899d-5e8158d2638c-kube-api-access-99s4b\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:53.692746 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.692486    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:53.692746 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.692501    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5bec735-88e4-4b14-899d-5e8158d2638c-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:53.692746 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.692517    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5bec735-88e4-4b14-899d-5e8158d2638c-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:53.919624 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.919520    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"]
Oct 30 01:19:53.930209 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:53.930177    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq"]
Oct 30 01:19:54.372198 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:54.372165    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8s5bq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:19:54.372382 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:54.372233    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8s5bq" podUID="b5bec735-88e4-4b14-899d-5e8158d2638c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:19:54.384392 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:54.384365    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"]
Oct 30 01:19:54.384636 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:54.384575    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54" podUID="a4ebf7d4-a370-441f-88c3-10c9ff7ee772" containerName="istio-proxy" containerID="cri-o://2325cabd837a18ecc0447f0c4aff7cf71a0cb8d1d80608c6ad329f45a3fbff73" gracePeriod=30
Oct 30 01:19:54.394983 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:54.394955    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"]
Oct 30 01:19:54.395242 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:54.395213    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j" podUID="0838696e-e4dd-4f9f-90b8-579c9955ac74" containerName="istio-proxy" containerID="cri-o://09db19c32d2d80a19eaf88774e9d04ce9e23b569a63572a941fa669261839f17" gracePeriod=30
Oct 30 01:19:55.794052 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:55.794017    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5bec735-88e4-4b14-899d-5e8158d2638c" path="/var/lib/kubelet/pods/b5bec735-88e4-4b14-899d-5e8158d2638c/volumes"
Oct 30 01:19:59.636563 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.636530    4864 generic.go:356] "Generic (PLEG): container finished" podID="0838696e-e4dd-4f9f-90b8-579c9955ac74" containerID="09db19c32d2d80a19eaf88774e9d04ce9e23b569a63572a941fa669261839f17" exitCode=0
Oct 30 01:19:59.636978 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.636654    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j" event={"ID":"0838696e-e4dd-4f9f-90b8-579c9955ac74","Type":"ContainerDied","Data":"09db19c32d2d80a19eaf88774e9d04ce9e23b569a63572a941fa669261839f17"}
Oct 30 01:19:59.638561 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.638533    4864 generic.go:356] "Generic (PLEG): container finished" podID="a4ebf7d4-a370-441f-88c3-10c9ff7ee772" containerID="2325cabd837a18ecc0447f0c4aff7cf71a0cb8d1d80608c6ad329f45a3fbff73" exitCode=0
Oct 30 01:19:59.638707 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.638613    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54" event={"ID":"a4ebf7d4-a370-441f-88c3-10c9ff7ee772","Type":"ContainerDied","Data":"2325cabd837a18ecc0447f0c4aff7cf71a0cb8d1d80608c6ad329f45a3fbff73"}
Oct 30 01:19:59.674997 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.674968    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:19:59.719278 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.719250    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:19:59.795991 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.795953    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-workload-socket\") pod \"0838696e-e4dd-4f9f-90b8-579c9955ac74\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") "
Oct 30 01:19:59.796210 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.796058    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0838696e-e4dd-4f9f-90b8-579c9955ac74-kube-api-access-swjvn\" (UniqueName: \"kubernetes.io/projected/0838696e-e4dd-4f9f-90b8-579c9955ac74-kube-api-access-swjvn\") pod \"0838696e-e4dd-4f9f-90b8-579c9955ac74\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") "
Oct 30 01:19:59.796210 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.796143    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-workload-certs\") pod \"0838696e-e4dd-4f9f-90b8-579c9955ac74\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") "
Oct 30 01:19:59.796396 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.796345    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-workload-socket" pod "0838696e-e4dd-4f9f-90b8-579c9955ac74" (UID: "0838696e-e4dd-4f9f-90b8-579c9955ac74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:59.796396 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.796365    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-data\") pod \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") "
Oct 30 01:19:59.796517 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.796430    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-kube-api-access-tkbrj\" (UniqueName: \"kubernetes.io/projected/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-kube-api-access-tkbrj\") pod \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") "
Oct 30 01:19:59.796517 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.796482    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-workload-certs" pod "0838696e-e4dd-4f9f-90b8-579c9955ac74" (UID: "0838696e-e4dd-4f9f-90b8-579c9955ac74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:59.796517 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.796502    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0838696e-e4dd-4f9f-90b8-579c9955ac74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0838696e-e4dd-4f9f-90b8-579c9955ac74-istiod-ca-cert\") pod \"0838696e-e4dd-4f9f-90b8-579c9955ac74\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") "
Oct 30 01:19:59.796670 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.796552    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istiod-ca-cert\") pod \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") "
Oct 30 01:19:59.796724 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.796667    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-envoy\") pod \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") "
Oct 30 01:19:59.796724 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.796713    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-credential-socket\") pod \"0838696e-e4dd-4f9f-90b8-579c9955ac74\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") "
Oct 30 01:19:59.796826 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.796757    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-podinfo\") pod \"0838696e-e4dd-4f9f-90b8-579c9955ac74\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") "
Oct 30 01:19:59.796826 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.796787    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-token\" (UniqueName: \"kubernetes.io/projected/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-token\") pod \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") "
Oct 30 01:19:59.796925 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.796834    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-envoy\") pod \"0838696e-e4dd-4f9f-90b8-579c9955ac74\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") "
Oct 30 01:19:59.796925 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.796888    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-data\") pod \"0838696e-e4dd-4f9f-90b8-579c9955ac74\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") "
Oct 30 01:19:59.797083 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.796931    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-workload-socket\") pod \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") "
Oct 30 01:19:59.797083 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.796950    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0838696e-e4dd-4f9f-90b8-579c9955ac74-istiod-ca-cert" pod "0838696e-e4dd-4f9f-90b8-579c9955ac74" (UID: "0838696e-e4dd-4f9f-90b8-579c9955ac74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:59.797083 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.796966    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-workload-certs\") pod \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") "
Oct 30 01:19:59.797083 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.797011    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-token\" (UniqueName: \"kubernetes.io/projected/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-token\") pod \"0838696e-e4dd-4f9f-90b8-579c9955ac74\" (UID: \"0838696e-e4dd-4f9f-90b8-579c9955ac74\") "
Oct 30 01:19:59.797083 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.797049    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-credential-socket\") pod \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") "
Oct 30 01:19:59.797329 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.797116    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-podinfo\") pod \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\" (UID: \"a4ebf7d4-a370-441f-88c3-10c9ff7ee772\") "
Oct 30 01:19:59.797726 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.797704    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.797825 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.797731    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.797825 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.797749    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0838696e-e4dd-4f9f-90b8-579c9955ac74-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.798925 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.798386    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istiod-ca-cert" pod "a4ebf7d4-a370-441f-88c3-10c9ff7ee772" (UID: "a4ebf7d4-a370-441f-88c3-10c9ff7ee772"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:59.798925 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.798686    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-credential-socket" pod "0838696e-e4dd-4f9f-90b8-579c9955ac74" (UID: "0838696e-e4dd-4f9f-90b8-579c9955ac74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:59.798925 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.798799    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-credential-socket" pod "a4ebf7d4-a370-441f-88c3-10c9ff7ee772" (UID: "a4ebf7d4-a370-441f-88c3-10c9ff7ee772"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:59.799167 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.798964    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-data" pod "0838696e-e4dd-4f9f-90b8-579c9955ac74" (UID: "0838696e-e4dd-4f9f-90b8-579c9955ac74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:59.799679 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.799307    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-workload-socket" pod "a4ebf7d4-a370-441f-88c3-10c9ff7ee772" (UID: "a4ebf7d4-a370-441f-88c3-10c9ff7ee772"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:59.799679 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.799425    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-workload-certs" pod "a4ebf7d4-a370-441f-88c3-10c9ff7ee772" (UID: "a4ebf7d4-a370-441f-88c3-10c9ff7ee772"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:59.800006 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.799979    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0838696e-e4dd-4f9f-90b8-579c9955ac74-kube-api-access-swjvn" pod "0838696e-e4dd-4f9f-90b8-579c9955ac74" (UID: "0838696e-e4dd-4f9f-90b8-579c9955ac74"). InnerVolumeSpecName "kube-api-access-swjvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:59.800300 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.800260    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-podinfo" pod "a4ebf7d4-a370-441f-88c3-10c9ff7ee772" (UID: "a4ebf7d4-a370-441f-88c3-10c9ff7ee772"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:19:59.800582 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.800508    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-podinfo" pod "0838696e-e4dd-4f9f-90b8-579c9955ac74" (UID: "0838696e-e4dd-4f9f-90b8-579c9955ac74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:19:59.801016 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.800967    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-data" pod "a4ebf7d4-a370-441f-88c3-10c9ff7ee772" (UID: "a4ebf7d4-a370-441f-88c3-10c9ff7ee772"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:59.801497 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.801461    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-kube-api-access-tkbrj" pod "a4ebf7d4-a370-441f-88c3-10c9ff7ee772" (UID: "a4ebf7d4-a370-441f-88c3-10c9ff7ee772"). InnerVolumeSpecName "kube-api-access-tkbrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:59.802285 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.802256    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-token" pod "0838696e-e4dd-4f9f-90b8-579c9955ac74" (UID: "0838696e-e4dd-4f9f-90b8-579c9955ac74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:59.802396 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.802371    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-token" pod "a4ebf7d4-a370-441f-88c3-10c9ff7ee772" (UID: "a4ebf7d4-a370-441f-88c3-10c9ff7ee772"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:59.802465 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.802430    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-envoy" pod "a4ebf7d4-a370-441f-88c3-10c9ff7ee772" (UID: "a4ebf7d4-a370-441f-88c3-10c9ff7ee772"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:59.802685 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.802664    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-envoy" pod "0838696e-e4dd-4f9f-90b8-579c9955ac74" (UID: "0838696e-e4dd-4f9f-90b8-579c9955ac74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:59.899362 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.899275    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.899362 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.899306    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkbrj\" (UniqueName: \"kubernetes.io/projected/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-kube-api-access-tkbrj\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.899362 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.899317    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.899362 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.899326    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.899362 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.899334    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.899362 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.899343    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.899362 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.899351    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.899362 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.899359    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.899362 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.899366    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.899780 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.899374    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.899780 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.899382    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.899780 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.899390    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0838696e-e4dd-4f9f-90b8-579c9955ac74-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.899780 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.899397    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.899780 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.899405    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ebf7d4-a370-441f-88c3-10c9ff7ee772-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.899780 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:19:59.899413    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swjvn\" (UniqueName: \"kubernetes.io/projected/0838696e-e4dd-4f9f-90b8-579c9955ac74-kube-api-access-swjvn\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:20:00.646456 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:00.646430    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"
Oct 30 01:20:00.646930 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:00.646451    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j" event={"ID":"0838696e-e4dd-4f9f-90b8-579c9955ac74","Type":"ContainerDied","Data":"64ef7e2da7ff83e1c282e622c902a9b45ec75d31408a3c2c84c42022e9af3dd7"}
Oct 30 01:20:00.646930 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:00.646520    4864 scope.go:122] "RemoveContainer" containerID="09db19c32d2d80a19eaf88774e9d04ce9e23b569a63572a941fa669261839f17"
Oct 30 01:20:00.648115 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:00.648093    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"
Oct 30 01:20:00.648270 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:00.648067    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54" event={"ID":"a4ebf7d4-a370-441f-88c3-10c9ff7ee772","Type":"ContainerDied","Data":"b737380a6ec10440329dc5dade1a32a5c3c7818009e997baa3e93d9003c7bb9d"}
Oct 30 01:20:00.696808 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:00.696785    4864 scope.go:122] "RemoveContainer" containerID="2325cabd837a18ecc0447f0c4aff7cf71a0cb8d1d80608c6ad329f45a3fbff73"
Oct 30 01:20:00.739206 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:00.739171    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"]
Oct 30 01:20:00.748168 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:00.748142    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4mp54"]
Oct 30 01:20:00.774223 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:00.774194    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"]
Oct 30 01:20:00.778731 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:00.778694    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-q6j5j"]
Oct 30 01:20:01.794310 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:01.794273    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0838696e-e4dd-4f9f-90b8-579c9955ac74" path="/var/lib/kubelet/pods/0838696e-e4dd-4f9f-90b8-579c9955ac74/volumes"
Oct 30 01:20:01.794800 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:01.794785    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4ebf7d4-a370-441f-88c3-10c9ff7ee772" path="/var/lib/kubelet/pods/a4ebf7d4-a370-441f-88c3-10c9ff7ee772/volumes"
Oct 30 01:20:19.040545 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:20:19.040502    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:20:22.449173 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:22.449137    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"]
Oct 30 01:20:22.449655 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:22.449430    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj" podUID="e5461f8d-22e4-4d42-b19b-9607ddbd97a0" containerName="istio-proxy" containerID="cri-o://94c7a9e0f554e81d61aa4edcf3c0e3e31e0673d2c5d26421715af3faa51660c8" gracePeriod=30
Oct 30 01:20:26.789647 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:26.789608    4864 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:20:26.790240 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:26.789682    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj" podUID="e5461f8d-22e4-4d42-b19b-9607ddbd97a0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:20:27.712814 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.712786    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 01:20:27.792049 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.791960    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istiod-ca-cert\") pod \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") "
Oct 30 01:20:27.792049 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.792028    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-workload-socket\") pod \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") "
Oct 30 01:20:27.792621 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.792088    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-token\" (UniqueName: \"kubernetes.io/projected/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-token\") pod \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") "
Oct 30 01:20:27.792621 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.792127    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-data\") pod \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") "
Oct 30 01:20:27.792621 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.792186    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-kube-api-access-gc9v5\" (UniqueName: \"kubernetes.io/projected/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-kube-api-access-gc9v5\") pod \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") "
Oct 30 01:20:27.792621 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.792233    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-credential-socket\") pod \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") "
Oct 30 01:20:27.792621 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.792303    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-envoy\") pod \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") "
Oct 30 01:20:27.792621 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.792373    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-workload-certs\") pod \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") "
Oct 30 01:20:27.792621 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.792400    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-podinfo\") pod \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\" (UID: \"e5461f8d-22e4-4d42-b19b-9607ddbd97a0\") "
Oct 30 01:20:27.792621 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.792413    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istiod-ca-cert" pod "e5461f8d-22e4-4d42-b19b-9607ddbd97a0" (UID: "e5461f8d-22e4-4d42-b19b-9607ddbd97a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:27.793065 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.792776    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-workload-socket" pod "e5461f8d-22e4-4d42-b19b-9607ddbd97a0" (UID: "e5461f8d-22e4-4d42-b19b-9607ddbd97a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:27.793150 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.793117    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-credential-socket" pod "e5461f8d-22e4-4d42-b19b-9607ddbd97a0" (UID: "e5461f8d-22e4-4d42-b19b-9607ddbd97a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:27.793253 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.793230    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:20:27.793403 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.793380    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-workload-certs" pod "e5461f8d-22e4-4d42-b19b-9607ddbd97a0" (UID: "e5461f8d-22e4-4d42-b19b-9607ddbd97a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:27.793501 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.793485    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:20:27.793670 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.793648    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-data" pod "e5461f8d-22e4-4d42-b19b-9607ddbd97a0" (UID: "e5461f8d-22e4-4d42-b19b-9607ddbd97a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:27.795012 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.794985    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-podinfo" pod "e5461f8d-22e4-4d42-b19b-9607ddbd97a0" (UID: "e5461f8d-22e4-4d42-b19b-9607ddbd97a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:20:27.795472 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.795435    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-envoy" pod "e5461f8d-22e4-4d42-b19b-9607ddbd97a0" (UID: "e5461f8d-22e4-4d42-b19b-9607ddbd97a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:27.795585 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.795565    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-token" pod "e5461f8d-22e4-4d42-b19b-9607ddbd97a0" (UID: "e5461f8d-22e4-4d42-b19b-9607ddbd97a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:27.796761 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.796734    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-kube-api-access-gc9v5" pod "e5461f8d-22e4-4d42-b19b-9607ddbd97a0" (UID: "e5461f8d-22e4-4d42-b19b-9607ddbd97a0"). InnerVolumeSpecName "kube-api-access-gc9v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:27.866601 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.866568    4864 generic.go:356] "Generic (PLEG): container finished" podID="e5461f8d-22e4-4d42-b19b-9607ddbd97a0" containerID="94c7a9e0f554e81d61aa4edcf3c0e3e31e0673d2c5d26421715af3faa51660c8" exitCode=0
Oct 30 01:20:27.866765 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.866632    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj" event={"ID":"e5461f8d-22e4-4d42-b19b-9607ddbd97a0","Type":"ContainerDied","Data":"94c7a9e0f554e81d61aa4edcf3c0e3e31e0673d2c5d26421715af3faa51660c8"}
Oct 30 01:20:27.866765 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.866634    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"
Oct 30 01:20:27.866765 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.866676    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj" event={"ID":"e5461f8d-22e4-4d42-b19b-9607ddbd97a0","Type":"ContainerDied","Data":"5f61d4ccd3c7364399c3bc0b49b8ca648c0ad9d032131fb4dd64f6d82d9c86b6"}
Oct 30 01:20:27.866765 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.866697    4864 scope.go:122] "RemoveContainer" containerID="94c7a9e0f554e81d61aa4edcf3c0e3e31e0673d2c5d26421715af3faa51660c8"
Oct 30 01:20:27.892502 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.892470    4864 scope.go:122] "RemoveContainer" containerID="94c7a9e0f554e81d61aa4edcf3c0e3e31e0673d2c5d26421715af3faa51660c8"
Oct 30 01:20:27.892831 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:20:27.892809    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c7a9e0f554e81d61aa4edcf3c0e3e31e0673d2c5d26421715af3faa51660c8\": container with ID starting with 94c7a9e0f554e81d61aa4edcf3c0e3e31e0673d2c5d26421715af3faa51660c8 not found: ID does not exist" containerID="94c7a9e0f554e81d61aa4edcf3c0e3e31e0673d2c5d26421715af3faa51660c8"
Oct 30 01:20:27.892885 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.892841    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c7a9e0f554e81d61aa4edcf3c0e3e31e0673d2c5d26421715af3faa51660c8"} err="failed to get container status \"94c7a9e0f554e81d61aa4edcf3c0e3e31e0673d2c5d26421715af3faa51660c8\": rpc error: code = NotFound desc = could not find container \"94c7a9e0f554e81d61aa4edcf3c0e3e31e0673d2c5d26421715af3faa51660c8\": container with ID starting with 94c7a9e0f554e81d61aa4edcf3c0e3e31e0673d2c5d26421715af3faa51660c8 not found: ID does not exist"
Oct 30 01:20:27.894613 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.894592    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:20:27.894613 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.894612    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:20:27.894757 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.894621    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc9v5\" (UniqueName: \"kubernetes.io/projected/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-kube-api-access-gc9v5\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:20:27.894757 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.894631    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:20:27.894757 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.894640    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:20:27.894757 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.894648    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:20:27.894757 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.894655    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5461f8d-22e4-4d42-b19b-9607ddbd97a0-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:20:27.921003 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.920971    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"]
Oct 30 01:20:27.924665 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:27.924634    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-f4rdj"]
Oct 30 01:20:28.613903 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.613863    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s"]
Oct 30 01:20:28.614603 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.614582    4864 cpu_manager.go:414] "containerMap: removing container" podUID="0dbf1a8d-c934-43fa-8a04-812019a09865" containerName="main"
Oct 30 01:20:28.614603 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.614603    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dbf1a8d-c934-43fa-8a04-812019a09865" containerName="main"
Oct 30 01:20:28.614711 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.614622    4864 cpu_manager.go:414] "containerMap: removing container" podUID="dc29530c-080c-4698-a2f3-7300a77e9804" containerName="istio-proxy"
Oct 30 01:20:28.614711 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.614627    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc29530c-080c-4698-a2f3-7300a77e9804" containerName="istio-proxy"
Oct 30 01:20:28.614711 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.614634    4864 cpu_manager.go:414] "containerMap: removing container" podUID="a4ebf7d4-a370-441f-88c3-10c9ff7ee772" containerName="istio-proxy"
Oct 30 01:20:28.614711 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.614639    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4ebf7d4-a370-441f-88c3-10c9ff7ee772" containerName="istio-proxy"
Oct 30 01:20:28.614711 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.614652    4864 cpu_manager.go:414] "containerMap: removing container" podUID="b5bec735-88e4-4b14-899d-5e8158d2638c" containerName="istio-proxy"
Oct 30 01:20:28.614711 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.614658    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5bec735-88e4-4b14-899d-5e8158d2638c" containerName="istio-proxy"
Oct 30 01:20:28.614711 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.614669    4864 cpu_manager.go:414] "containerMap: removing container" podUID="0838696e-e4dd-4f9f-90b8-579c9955ac74" containerName="istio-proxy"
Oct 30 01:20:28.614711 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.614673    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0838696e-e4dd-4f9f-90b8-579c9955ac74" containerName="istio-proxy"
Oct 30 01:20:28.614711 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.614686    4864 cpu_manager.go:414] "containerMap: removing container" podUID="e5461f8d-22e4-4d42-b19b-9607ddbd97a0" containerName="istio-proxy"
Oct 30 01:20:28.614711 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.614691    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5461f8d-22e4-4d42-b19b-9607ddbd97a0" containerName="istio-proxy"
Oct 30 01:20:28.615220 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.614780    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5bec735-88e4-4b14-899d-5e8158d2638c" containerName="istio-proxy"
Oct 30 01:20:28.615220 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.614789    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="0838696e-e4dd-4f9f-90b8-579c9955ac74" containerName="istio-proxy"
Oct 30 01:20:28.615220 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.614798    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5461f8d-22e4-4d42-b19b-9607ddbd97a0" containerName="istio-proxy"
Oct 30 01:20:28.615220 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.614808    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dbf1a8d-c934-43fa-8a04-812019a09865" containerName="main"
Oct 30 01:20:28.615220 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.614815    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc29530c-080c-4698-a2f3-7300a77e9804" containerName="istio-proxy"
Oct 30 01:20:28.615220 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.614820    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4ebf7d4-a370-441f-88c3-10c9ff7ee772" containerName="istio-proxy"
Oct 30 01:20:28.619983 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.619964    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s"
Oct 30 01:20:28.634341 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.634315    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-bc127f\""
Oct 30 01:20:28.639157 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.639129    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s"]
Oct 30 01:20:28.669695 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.669662    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx"]
Oct 30 01:20:28.673673 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.673651    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx"
Oct 30 01:20:28.689996 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.689968    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-bc127f-dockercfg-cwvh4\""
Oct 30 01:20:28.693529 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.693504    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx"]
Oct 30 01:20:28.708253 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.708214    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef300502-05e8-43c9-99e3-657417491be0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s\" (UID: \"ef300502-05e8-43c9-99e3-657417491be0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s"
Oct 30 01:20:28.708432 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.708271    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpp79\" (UniqueName: \"kubernetes.io/projected/ef300502-05e8-43c9-99e3-657417491be0-kube-api-access-hpp79\") pod \"payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s\" (UID: \"ef300502-05e8-43c9-99e3-657417491be0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s"
Oct 30 01:20:28.708503 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.708427    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cx76\" (UniqueName: \"kubernetes.io/projected/90c062b6-b706-4e09-92a2-5627352f1485-kube-api-access-2cx76\") pod \"payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx\" (UID: \"90c062b6-b706-4e09-92a2-5627352f1485\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx"
Oct 30 01:20:28.708649 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.708624    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c062b6-b706-4e09-92a2-5627352f1485-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx\" (UID: \"90c062b6-b706-4e09-92a2-5627352f1485\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx"
Oct 30 01:20:28.813145 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.812615    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef300502-05e8-43c9-99e3-657417491be0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s\" (UID: \"ef300502-05e8-43c9-99e3-657417491be0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s"
Oct 30 01:20:28.813145 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.812680    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpp79\" (UniqueName: \"kubernetes.io/projected/ef300502-05e8-43c9-99e3-657417491be0-kube-api-access-hpp79\") pod \"payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s\" (UID: \"ef300502-05e8-43c9-99e3-657417491be0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s"
Oct 30 01:20:28.813145 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.812729    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cx76\" (UniqueName: \"kubernetes.io/projected/90c062b6-b706-4e09-92a2-5627352f1485-kube-api-access-2cx76\") pod \"payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx\" (UID: \"90c062b6-b706-4e09-92a2-5627352f1485\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx"
Oct 30 01:20:28.813145 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.812868    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c062b6-b706-4e09-92a2-5627352f1485-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx\" (UID: \"90c062b6-b706-4e09-92a2-5627352f1485\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx"
Oct 30 01:20:28.813796 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.813771    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c062b6-b706-4e09-92a2-5627352f1485-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx\" (UID: \"90c062b6-b706-4e09-92a2-5627352f1485\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx"
Oct 30 01:20:28.825107 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.821713    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef300502-05e8-43c9-99e3-657417491be0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s\" (UID: \"ef300502-05e8-43c9-99e3-657417491be0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s"
Oct 30 01:20:28.859477 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.859440    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpp79\" (UniqueName: \"kubernetes.io/projected/ef300502-05e8-43c9-99e3-657417491be0-kube-api-access-hpp79\") pod \"payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s\" (UID: \"ef300502-05e8-43c9-99e3-657417491be0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s"
Oct 30 01:20:28.864492 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.864457    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cx76\" (UniqueName: \"kubernetes.io/projected/90c062b6-b706-4e09-92a2-5627352f1485-kube-api-access-2cx76\") pod \"payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx\" (UID: \"90c062b6-b706-4e09-92a2-5627352f1485\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx"
Oct 30 01:20:28.934677 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.934644    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s"
Oct 30 01:20:28.989456 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:28.989419    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx"
Oct 30 01:20:29.285169 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:29.285136    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s"]
Oct 30 01:20:29.286219 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:20:29.286185    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef300502_05e8_43c9_99e3_657417491be0.slice/crio-8d2e4ae5bb8492c36604a92ed52a6447b5860b7a34a2e9cda2eac80e0732352f WatchSource:0}: Error finding container 8d2e4ae5bb8492c36604a92ed52a6447b5860b7a34a2e9cda2eac80e0732352f: Status 404 returned error can't find the container with id 8d2e4ae5bb8492c36604a92ed52a6447b5860b7a34a2e9cda2eac80e0732352f
Oct 30 01:20:29.337239 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:29.337214    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx"]
Oct 30 01:20:29.342371 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:20:29.342344    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c062b6_b706_4e09_92a2_5627352f1485.slice/crio-fbb613556b33e0ae42e8757a647952792a8386b8642889e68b08862d7d3ce31e WatchSource:0}: Error finding container fbb613556b33e0ae42e8757a647952792a8386b8642889e68b08862d7d3ce31e: Status 404 returned error can't find the container with id fbb613556b33e0ae42e8757a647952792a8386b8642889e68b08862d7d3ce31e
Oct 30 01:20:29.793975 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:29.793942    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5461f8d-22e4-4d42-b19b-9607ddbd97a0" path="/var/lib/kubelet/pods/e5461f8d-22e4-4d42-b19b-9607ddbd97a0/volumes"
Oct 30 01:20:29.855773 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:29.855735    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"]
Oct 30 01:20:29.856256 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:29.855947    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc" podUID="8e81471d-5cf3-4486-b5bf-54fb0a99702d" containerName="istio-proxy" containerID="cri-o://4d510d7637350af0e99fc4a5872eb53874e2e2b383c9cd9770fa78247c658c17" gracePeriod=30
Oct 30 01:20:29.889307 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:29.889233    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s" event={"ID":"ef300502-05e8-43c9-99e3-657417491be0","Type":"ContainerStarted","Data":"e9157a2510958064632209186ac8bf11b21bc9ef474a3fca62c6fc3d57cd0ad0"}
Oct 30 01:20:29.889307 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:29.889308    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s"
Oct 30 01:20:29.889565 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:29.889342    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s" event={"ID":"ef300502-05e8-43c9-99e3-657417491be0","Type":"ContainerStarted","Data":"8d2e4ae5bb8492c36604a92ed52a6447b5860b7a34a2e9cda2eac80e0732352f"}
Oct 30 01:20:29.891462 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:29.891427    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx" event={"ID":"90c062b6-b706-4e09-92a2-5627352f1485","Type":"ContainerStarted","Data":"84e6f6ebb1a018a207e251ccbd71e921296bc52d655fbee1d2424625c62899a1"}
Oct 30 01:20:29.891593 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:29.891469    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx" event={"ID":"90c062b6-b706-4e09-92a2-5627352f1485","Type":"ContainerStarted","Data":"fbb613556b33e0ae42e8757a647952792a8386b8642889e68b08862d7d3ce31e"}
Oct 30 01:20:29.891593 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:29.891495    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx"
Oct 30 01:20:29.955671 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:29.955616    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s" podStartSLOduration=1.95560073 podStartE2EDuration="1.95560073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:29.953038825 +0000 UTC m=+2342.291140735" watchObservedRunningTime="2026-10-30 01:20:29.95560073 +0000 UTC m=+2342.293702640"
Oct 30 01:20:29.993617 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:29.993554    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx" podStartSLOduration=1.993540872 podStartE2EDuration="1.993540872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:29.991512393 +0000 UTC m=+2342.329614304" watchObservedRunningTime="2026-10-30 01:20:29.993540872 +0000 UTC m=+2342.331642826"
Oct 30 01:20:35.138416 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.138349    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:20:35.206548 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.206512    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-token\" (UniqueName: \"kubernetes.io/projected/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-token\") pod \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") "
Oct 30 01:20:35.206744 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.206573    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-workload-socket\") pod \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") "
Oct 30 01:20:35.206744 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.206604    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-podinfo\") pod \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") "
Oct 30 01:20:35.206744 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.206652    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-credential-socket\") pod \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") "
Oct 30 01:20:35.206744 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.206701    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-data\") pod \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") "
Oct 30 01:20:35.206978 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.206771    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istiod-ca-cert\") pod \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") "
Oct 30 01:20:35.206978 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.206813    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-envoy\") pod \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") "
Oct 30 01:20:35.206978 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.206900    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-workload-certs\") pod \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") "
Oct 30 01:20:35.207154 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.207008    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-credential-socket" pod "8e81471d-5cf3-4486-b5bf-54fb0a99702d" (UID: "8e81471d-5cf3-4486-b5bf-54fb0a99702d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:35.207348 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.207323    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e81471d-5cf3-4486-b5bf-54fb0a99702d-kube-api-access-m878p\" (UniqueName: \"kubernetes.io/projected/8e81471d-5cf3-4486-b5bf-54fb0a99702d-kube-api-access-m878p\") pod \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\" (UID: \"8e81471d-5cf3-4486-b5bf-54fb0a99702d\") "
Oct 30 01:20:35.207427 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.207350    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-workload-certs" pod "8e81471d-5cf3-4486-b5bf-54fb0a99702d" (UID: "8e81471d-5cf3-4486-b5bf-54fb0a99702d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:35.207633 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.207609    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istiod-ca-cert" pod "8e81471d-5cf3-4486-b5bf-54fb0a99702d" (UID: "8e81471d-5cf3-4486-b5bf-54fb0a99702d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:35.208038 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.208014    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:20:35.208148 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.208043    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:20:35.208148 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.208058    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:20:35.208496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.208470    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-data" pod "8e81471d-5cf3-4486-b5bf-54fb0a99702d" (UID: "8e81471d-5cf3-4486-b5bf-54fb0a99702d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:35.208609 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.208531    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-workload-socket" pod "8e81471d-5cf3-4486-b5bf-54fb0a99702d" (UID: "8e81471d-5cf3-4486-b5bf-54fb0a99702d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:35.209791 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.209766    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e81471d-5cf3-4486-b5bf-54fb0a99702d-kube-api-access-m878p" pod "8e81471d-5cf3-4486-b5bf-54fb0a99702d" (UID: "8e81471d-5cf3-4486-b5bf-54fb0a99702d"). InnerVolumeSpecName "kube-api-access-m878p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:35.210386 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.210366    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-envoy" pod "8e81471d-5cf3-4486-b5bf-54fb0a99702d" (UID: "8e81471d-5cf3-4486-b5bf-54fb0a99702d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:35.211432 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.211407    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-token" pod "8e81471d-5cf3-4486-b5bf-54fb0a99702d" (UID: "8e81471d-5cf3-4486-b5bf-54fb0a99702d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:35.212560 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.212535    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-podinfo" pod "8e81471d-5cf3-4486-b5bf-54fb0a99702d" (UID: "8e81471d-5cf3-4486-b5bf-54fb0a99702d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:20:35.309932 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.309893    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m878p\" (UniqueName: \"kubernetes.io/projected/8e81471d-5cf3-4486-b5bf-54fb0a99702d-kube-api-access-m878p\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:20:35.309932 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.309933    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:20:35.310183 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.309949    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:20:35.310183 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.309963    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:20:35.310183 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.309976    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:20:35.310183 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.309988    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e81471d-5cf3-4486-b5bf-54fb0a99702d-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:20:35.933298 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.933263    4864 generic.go:356] "Generic (PLEG): container finished" podID="8e81471d-5cf3-4486-b5bf-54fb0a99702d" containerID="4d510d7637350af0e99fc4a5872eb53874e2e2b383c9cd9770fa78247c658c17" exitCode=0
Oct 30 01:20:35.933469 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.933316    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"
Oct 30 01:20:35.933469 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.933351    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc" event={"ID":"8e81471d-5cf3-4486-b5bf-54fb0a99702d","Type":"ContainerDied","Data":"4d510d7637350af0e99fc4a5872eb53874e2e2b383c9cd9770fa78247c658c17"}
Oct 30 01:20:35.933469 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.933410    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc" event={"ID":"8e81471d-5cf3-4486-b5bf-54fb0a99702d","Type":"ContainerDied","Data":"376f1e969dbb92c2d4115a43173bec9419d4f18a2fed14aab9c994b6ecf83e06"}
Oct 30 01:20:35.933469 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.933431    4864 scope.go:122] "RemoveContainer" containerID="4d510d7637350af0e99fc4a5872eb53874e2e2b383c9cd9770fa78247c658c17"
Oct 30 01:20:35.943100 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.943059    4864 scope.go:122] "RemoveContainer" containerID="4d510d7637350af0e99fc4a5872eb53874e2e2b383c9cd9770fa78247c658c17"
Oct 30 01:20:35.943389 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:20:35.943363    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d510d7637350af0e99fc4a5872eb53874e2e2b383c9cd9770fa78247c658c17\": container with ID starting with 4d510d7637350af0e99fc4a5872eb53874e2e2b383c9cd9770fa78247c658c17 not found: ID does not exist" containerID="4d510d7637350af0e99fc4a5872eb53874e2e2b383c9cd9770fa78247c658c17"
Oct 30 01:20:35.943507 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.943394    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d510d7637350af0e99fc4a5872eb53874e2e2b383c9cd9770fa78247c658c17"} err="failed to get container status \"4d510d7637350af0e99fc4a5872eb53874e2e2b383c9cd9770fa78247c658c17\": rpc error: code = NotFound desc = could not find container \"4d510d7637350af0e99fc4a5872eb53874e2e2b383c9cd9770fa78247c658c17\": container with ID starting with 4d510d7637350af0e99fc4a5872eb53874e2e2b383c9cd9770fa78247c658c17 not found: ID does not exist"
Oct 30 01:20:35.991690 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:35.991657    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"]
Oct 30 01:20:36.003697 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:36.003666    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-cbcqc"]
Oct 30 01:20:37.794424 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:37.794389    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e81471d-5cf3-4486-b5bf-54fb0a99702d" path="/var/lib/kubelet/pods/8e81471d-5cf3-4486-b5bf-54fb0a99702d/volumes"
Oct 30 01:20:40.930102 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:40.929240    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s"
Oct 30 01:20:40.936343 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:40.936243    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx"
Oct 30 01:20:44.515389 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:44.515355    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7291-3cb6d3"]
Oct 30 01:20:44.515937 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:44.515923    4864 cpu_manager.go:414] "containerMap: removing container" podUID="8e81471d-5cf3-4486-b5bf-54fb0a99702d" containerName="istio-proxy"
Oct 30 01:20:44.515981 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:44.515938    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e81471d-5cf3-4486-b5bf-54fb0a99702d" containerName="istio-proxy"
Oct 30 01:20:44.516053 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:44.516044    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e81471d-5cf3-4486-b5bf-54fb0a99702d" containerName="istio-proxy"
Oct 30 01:20:44.519403 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:44.519386    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7291-3cb6d3"
Oct 30 01:20:44.537280 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:44.537255    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7291-3cb6d3"]
Oct 30 01:20:44.543941 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:44.543915    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4z6vl\""
Oct 30 01:20:44.653451 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:44.653415    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gblz9\" (UniqueName: \"kubernetes.io/projected/242f0ad3-3304-4f18-aad2-90171bc84b68-kube-api-access-gblz9\") pod \"test-curl-7291-3cb6d3\" (UID: \"242f0ad3-3304-4f18-aad2-90171bc84b68\") " pod="opendatahub/test-curl-7291-3cb6d3"
Oct 30 01:20:44.754653 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:44.754620    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gblz9\" (UniqueName: \"kubernetes.io/projected/242f0ad3-3304-4f18-aad2-90171bc84b68-kube-api-access-gblz9\") pod \"test-curl-7291-3cb6d3\" (UID: \"242f0ad3-3304-4f18-aad2-90171bc84b68\") " pod="opendatahub/test-curl-7291-3cb6d3"
Oct 30 01:20:44.775789 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:44.775724    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gblz9\" (UniqueName: \"kubernetes.io/projected/242f0ad3-3304-4f18-aad2-90171bc84b68-kube-api-access-gblz9\") pod \"test-curl-7291-3cb6d3\" (UID: \"242f0ad3-3304-4f18-aad2-90171bc84b68\") " pod="opendatahub/test-curl-7291-3cb6d3"
Oct 30 01:20:44.831891 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:44.831858    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7291-3cb6d3"
Oct 30 01:20:45.127303 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:45.127276    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7291-3cb6d3"]
Oct 30 01:20:45.129007 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:20:45.128981    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod242f0ad3_3304_4f18_aad2_90171bc84b68.slice/crio-78e5b7c74c46cd672a22e4d3d66d14458bdd0071d3c7edf56899afccbf16ca90 WatchSource:0}: Error finding container 78e5b7c74c46cd672a22e4d3d66d14458bdd0071d3c7edf56899afccbf16ca90: Status 404 returned error can't find the container with id 78e5b7c74c46cd672a22e4d3d66d14458bdd0071d3c7edf56899afccbf16ca90
Oct 30 01:20:46.020193 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:46.020160    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7291-3cb6d3" event={"ID":"242f0ad3-3304-4f18-aad2-90171bc84b68","Type":"ContainerStarted","Data":"63ab3c042e19a38804ceca111f6a591d7f9639bc07e290f837a1ad7d451acfe6"}
Oct 30 01:20:46.020604 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:46.020201    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7291-3cb6d3" event={"ID":"242f0ad3-3304-4f18-aad2-90171bc84b68","Type":"ContainerStarted","Data":"78e5b7c74c46cd672a22e4d3d66d14458bdd0071d3c7edf56899afccbf16ca90"}
Oct 30 01:20:46.056567 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:46.056513    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7291-3cb6d3" podStartSLOduration=2.056498185 podStartE2EDuration="2.056498185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:46.053972056 +0000 UTC m=+2358.392073968" watchObservedRunningTime="2026-10-30 01:20:46.056498185 +0000 UTC m=+2358.394600094"
Oct 30 01:20:46.920825 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:46.920788    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7291-3cb6d3"]
Oct 30 01:20:46.928671 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:46.928641    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7291-3cb6d3"]
Oct 30 01:20:48.033291 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:48.033251    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7291-3cb6d3" podUID="242f0ad3-3304-4f18-aad2-90171bc84b68" containerName="test-curl-7291-3cb6d3" containerID="cri-o://63ab3c042e19a38804ceca111f6a591d7f9639bc07e290f837a1ad7d451acfe6" gracePeriod=2
Oct 30 01:20:49.044244 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:20:49.044209    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:20:50.252145 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:50.252119    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7291-3cb6d3"
Oct 30 01:20:50.343407 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:50.343312    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/242f0ad3-3304-4f18-aad2-90171bc84b68-kube-api-access-gblz9\" (UniqueName: \"kubernetes.io/projected/242f0ad3-3304-4f18-aad2-90171bc84b68-kube-api-access-gblz9\") pod \"242f0ad3-3304-4f18-aad2-90171bc84b68\" (UID: \"242f0ad3-3304-4f18-aad2-90171bc84b68\") "
Oct 30 01:20:50.345441 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:50.345408    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242f0ad3-3304-4f18-aad2-90171bc84b68-kube-api-access-gblz9" pod "242f0ad3-3304-4f18-aad2-90171bc84b68" (UID: "242f0ad3-3304-4f18-aad2-90171bc84b68"). InnerVolumeSpecName "kube-api-access-gblz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:50.444537 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:50.444502    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gblz9\" (UniqueName: \"kubernetes.io/projected/242f0ad3-3304-4f18-aad2-90171bc84b68-kube-api-access-gblz9\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:20:51.054832 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:51.054798    4864 generic.go:356] "Generic (PLEG): container finished" podID="242f0ad3-3304-4f18-aad2-90171bc84b68" containerID="63ab3c042e19a38804ceca111f6a591d7f9639bc07e290f837a1ad7d451acfe6" exitCode=137
Oct 30 01:20:51.055047 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:51.054843    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7291-3cb6d3"
Oct 30 01:20:51.055047 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:51.054900    4864 scope.go:122] "RemoveContainer" containerID="63ab3c042e19a38804ceca111f6a591d7f9639bc07e290f837a1ad7d451acfe6"
Oct 30 01:20:51.066198 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:51.066167    4864 scope.go:122] "RemoveContainer" containerID="63ab3c042e19a38804ceca111f6a591d7f9639bc07e290f837a1ad7d451acfe6"
Oct 30 01:20:51.066526 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:20:51.066501    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ab3c042e19a38804ceca111f6a591d7f9639bc07e290f837a1ad7d451acfe6\": container with ID starting with 63ab3c042e19a38804ceca111f6a591d7f9639bc07e290f837a1ad7d451acfe6 not found: ID does not exist" containerID="63ab3c042e19a38804ceca111f6a591d7f9639bc07e290f837a1ad7d451acfe6"
Oct 30 01:20:51.066624 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:51.066533    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ab3c042e19a38804ceca111f6a591d7f9639bc07e290f837a1ad7d451acfe6"} err="failed to get container status \"63ab3c042e19a38804ceca111f6a591d7f9639bc07e290f837a1ad7d451acfe6\": rpc error: code = NotFound desc = could not find container \"63ab3c042e19a38804ceca111f6a591d7f9639bc07e290f837a1ad7d451acfe6\": container with ID starting with 63ab3c042e19a38804ceca111f6a591d7f9639bc07e290f837a1ad7d451acfe6 not found: ID does not exist"
Oct 30 01:20:51.078343 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:20:51.078304    4864 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7291-3cb6d3\" is forbidden: User \"system:node:ip-10-0-134-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-210.ec2.internal' and this object" podUID="242f0ad3-3304-4f18-aad2-90171bc84b68" pod="opendatahub/test-curl-7291-3cb6d3"
Oct 30 01:20:51.793660 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:20:51.793617    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="242f0ad3-3304-4f18-aad2-90171bc84b68" path="/var/lib/kubelet/pods/242f0ad3-3304-4f18-aad2-90171bc84b68/volumes"
Oct 30 01:21:04.931635 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:04.931591    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7288-467c33"]
Oct 30 01:21:04.932507 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:04.932483    4864 cpu_manager.go:414] "containerMap: removing container" podUID="242f0ad3-3304-4f18-aad2-90171bc84b68" containerName="test-curl-7291-3cb6d3"
Oct 30 01:21:04.932615 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:04.932509    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="242f0ad3-3304-4f18-aad2-90171bc84b68" containerName="test-curl-7291-3cb6d3"
Oct 30 01:21:04.932679 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:04.932667    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="242f0ad3-3304-4f18-aad2-90171bc84b68" containerName="test-curl-7291-3cb6d3"
Oct 30 01:21:04.937305 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:04.937285    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7288-467c33"
Oct 30 01:21:04.953379 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:04.953352    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7288-467c33"]
Oct 30 01:21:04.959953 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:04.959929    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4z6vl\""
Oct 30 01:21:05.001639 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:05.001608    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ggk8\" (UniqueName: \"kubernetes.io/projected/24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d-kube-api-access-5ggk8\") pod \"test-curl-7288-467c33\" (UID: \"24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d\") " pod="opendatahub/test-curl-7288-467c33"
Oct 30 01:21:05.102382 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:05.102343    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ggk8\" (UniqueName: \"kubernetes.io/projected/24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d-kube-api-access-5ggk8\") pod \"test-curl-7288-467c33\" (UID: \"24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d\") " pod="opendatahub/test-curl-7288-467c33"
Oct 30 01:21:05.124362 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:05.124331    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ggk8\" (UniqueName: \"kubernetes.io/projected/24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d-kube-api-access-5ggk8\") pod \"test-curl-7288-467c33\" (UID: \"24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d\") " pod="opendatahub/test-curl-7288-467c33"
Oct 30 01:21:05.250210 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:05.250168    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7288-467c33"
Oct 30 01:21:05.583829 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:05.583747    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7288-467c33"]
Oct 30 01:21:05.588036 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:21:05.588007    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24ca63b9_fe42_4e3a_a3a1_7c58e9a68c4d.slice/crio-3973a31c78b57b7ca09f77813acfa42431620ad48275d373cfda11200b842fa0 WatchSource:0}: Error finding container 3973a31c78b57b7ca09f77813acfa42431620ad48275d373cfda11200b842fa0: Status 404 returned error can't find the container with id 3973a31c78b57b7ca09f77813acfa42431620ad48275d373cfda11200b842fa0
Oct 30 01:21:06.152828 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:06.152790    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7288-467c33" event={"ID":"24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d","Type":"ContainerStarted","Data":"6af92ecfd1b54f99985245c6d593cd129ed236381f54f8159536f82d8a7d0a4b"}
Oct 30 01:21:06.153247 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:06.152834    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7288-467c33" event={"ID":"24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d","Type":"ContainerStarted","Data":"3973a31c78b57b7ca09f77813acfa42431620ad48275d373cfda11200b842fa0"}
Oct 30 01:21:06.188344 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:06.188286    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7288-467c33" podStartSLOduration=2.188266191 podStartE2EDuration="2.188266191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:06.186818859 +0000 UTC m=+2378.524920770" watchObservedRunningTime="2026-10-30 01:21:06.188266191 +0000 UTC m=+2378.526368100"
Oct 30 01:21:07.054973 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:07.054930    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7288-467c33"]
Oct 30 01:21:07.060001 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:07.059967    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7288-467c33"]
Oct 30 01:21:07.198395 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:07.198361    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7288-7734ce"]
Oct 30 01:21:07.198989 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:07.198970    4864 cpu_manager.go:414] "containerMap: removing container" podUID="24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d" containerName="test-curl-7288-467c33"
Oct 30 01:21:07.199057 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:07.198989    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d" containerName="test-curl-7288-467c33"
Oct 30 01:21:07.199129 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:07.199092    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d" containerName="test-curl-7288-467c33"
Oct 30 01:21:07.202469 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:07.202451    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7288-7734ce"
Oct 30 01:21:07.219700 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:07.219670    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7288-7734ce"]
Oct 30 01:21:07.230268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:07.230235    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d5v8\" (UniqueName: \"kubernetes.io/projected/7f19e02b-f300-49b9-9c66-e8e04c3a860b-kube-api-access-2d5v8\") pod \"test-curl-7288-7734ce\" (UID: \"7f19e02b-f300-49b9-9c66-e8e04c3a860b\") " pod="opendatahub/test-curl-7288-7734ce"
Oct 30 01:21:07.331944 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:07.331867    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d5v8\" (UniqueName: \"kubernetes.io/projected/7f19e02b-f300-49b9-9c66-e8e04c3a860b-kube-api-access-2d5v8\") pod \"test-curl-7288-7734ce\" (UID: \"7f19e02b-f300-49b9-9c66-e8e04c3a860b\") " pod="opendatahub/test-curl-7288-7734ce"
Oct 30 01:21:07.356400 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:07.356371    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d5v8\" (UniqueName: \"kubernetes.io/projected/7f19e02b-f300-49b9-9c66-e8e04c3a860b-kube-api-access-2d5v8\") pod \"test-curl-7288-7734ce\" (UID: \"7f19e02b-f300-49b9-9c66-e8e04c3a860b\") " pod="opendatahub/test-curl-7288-7734ce"
Oct 30 01:21:07.515770 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:07.515741    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7288-7734ce"
Oct 30 01:21:07.822006 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:07.821973    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7288-7734ce"]
Oct 30 01:21:07.825844 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:21:07.825815    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f19e02b_f300_49b9_9c66_e8e04c3a860b.slice/crio-e9aa9c8e6d5c876ba13b500abbda4a368c7b4c80c18d3afbd6987f641e4a8f31 WatchSource:0}: Error finding container e9aa9c8e6d5c876ba13b500abbda4a368c7b4c80c18d3afbd6987f641e4a8f31: Status 404 returned error can't find the container with id e9aa9c8e6d5c876ba13b500abbda4a368c7b4c80c18d3afbd6987f641e4a8f31
Oct 30 01:21:08.166994 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:08.166900    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7288-7734ce" event={"ID":"7f19e02b-f300-49b9-9c66-e8e04c3a860b","Type":"ContainerStarted","Data":"5f64b76f28fd507a9f0a9d70732660e6ddcd05c1f03e826e5d00518bb441cbab"}
Oct 30 01:21:08.166994 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:08.166939    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7288-7734ce" event={"ID":"7f19e02b-f300-49b9-9c66-e8e04c3a860b","Type":"ContainerStarted","Data":"e9aa9c8e6d5c876ba13b500abbda4a368c7b4c80c18d3afbd6987f641e4a8f31"}
Oct 30 01:21:08.167263 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:08.167034    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7288-467c33" podUID="24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d" containerName="test-curl-7288-467c33" containerID="cri-o://6af92ecfd1b54f99985245c6d593cd129ed236381f54f8159536f82d8a7d0a4b" gracePeriod=2
Oct 30 01:21:08.202434 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:08.202374    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7288-7734ce" podStartSLOduration=1.202358577 podStartE2EDuration="1.202358577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:08.200655992 +0000 UTC m=+2380.538757916" watchObservedRunningTime="2026-10-30 01:21:08.202358577 +0000 UTC m=+2380.540460486"
Oct 30 01:21:09.139299 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:09.139258    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7288-7734ce"]
Oct 30 01:21:09.143553 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:09.143522    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7288-7734ce"]
Oct 30 01:21:09.295936 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:09.295896    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7288-b102db"]
Oct 30 01:21:09.296704 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:09.296681    4864 cpu_manager.go:414] "containerMap: removing container" podUID="7f19e02b-f300-49b9-9c66-e8e04c3a860b" containerName="test-curl-7288-7734ce"
Oct 30 01:21:09.296807 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:09.296705    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f19e02b-f300-49b9-9c66-e8e04c3a860b" containerName="test-curl-7288-7734ce"
Oct 30 01:21:09.296909 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:09.296896    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f19e02b-f300-49b9-9c66-e8e04c3a860b" containerName="test-curl-7288-7734ce"
Oct 30 01:21:09.300490 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:09.300468    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7288-b102db"
Oct 30 01:21:09.318422 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:09.318390    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7288-b102db"]
Oct 30 01:21:09.360772 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:09.360736    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97zs8\" (UniqueName: \"kubernetes.io/projected/cf146b7f-7e1c-497d-9a5d-20bff4bc411e-kube-api-access-97zs8\") pod \"test-curl-7288-b102db\" (UID: \"cf146b7f-7e1c-497d-9a5d-20bff4bc411e\") " pod="opendatahub/test-curl-7288-b102db"
Oct 30 01:21:09.462263 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:09.462224    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97zs8\" (UniqueName: \"kubernetes.io/projected/cf146b7f-7e1c-497d-9a5d-20bff4bc411e-kube-api-access-97zs8\") pod \"test-curl-7288-b102db\" (UID: \"cf146b7f-7e1c-497d-9a5d-20bff4bc411e\") " pod="opendatahub/test-curl-7288-b102db"
Oct 30 01:21:09.487594 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:09.487567    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97zs8\" (UniqueName: \"kubernetes.io/projected/cf146b7f-7e1c-497d-9a5d-20bff4bc411e-kube-api-access-97zs8\") pod \"test-curl-7288-b102db\" (UID: \"cf146b7f-7e1c-497d-9a5d-20bff4bc411e\") " pod="opendatahub/test-curl-7288-b102db"
Oct 30 01:21:09.613145 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:09.613114    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7288-b102db"
Oct 30 01:21:09.929103 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:09.929037    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7288-b102db"]
Oct 30 01:21:09.929977 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:21:09.929936    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf146b7f_7e1c_497d_9a5d_20bff4bc411e.slice/crio-0c6e6d7ca26d330114dbc4666ee81a7e8cbb55d43abf82a453f129d34f7342b1 WatchSource:0}: Error finding container 0c6e6d7ca26d330114dbc4666ee81a7e8cbb55d43abf82a453f129d34f7342b1: Status 404 returned error can't find the container with id 0c6e6d7ca26d330114dbc4666ee81a7e8cbb55d43abf82a453f129d34f7342b1
Oct 30 01:21:10.182964 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:10.182893    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7288-b102db" event={"ID":"cf146b7f-7e1c-497d-9a5d-20bff4bc411e","Type":"ContainerStarted","Data":"04ae1674caaad24c3d337ae03c803f4a4752679c238d109e34196bc564c27d2f"}
Oct 30 01:21:10.182964 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:10.182942    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7288-b102db" event={"ID":"cf146b7f-7e1c-497d-9a5d-20bff4bc411e","Type":"ContainerStarted","Data":"0c6e6d7ca26d330114dbc4666ee81a7e8cbb55d43abf82a453f129d34f7342b1"}
Oct 30 01:21:10.183190 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:10.183151    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7288-7734ce" podUID="7f19e02b-f300-49b9-9c66-e8e04c3a860b" containerName="test-curl-7288-7734ce" containerID="cri-o://5f64b76f28fd507a9f0a9d70732660e6ddcd05c1f03e826e5d00518bb441cbab" gracePeriod=2
Oct 30 01:21:10.225771 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:10.225713    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7288-b102db" podStartSLOduration=1.22569364 podStartE2EDuration="1.22569364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:10.222049894 +0000 UTC m=+2382.560151805" watchObservedRunningTime="2026-10-30 01:21:10.22569364 +0000 UTC m=+2382.563795552"
Oct 30 01:21:10.349352 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:10.349326    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7288-467c33"
Oct 30 01:21:10.374210 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:10.374179    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d-kube-api-access-5ggk8\" (UniqueName: \"kubernetes.io/projected/24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d-kube-api-access-5ggk8\") pod \"24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d\" (UID: \"24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d\") "
Oct 30 01:21:10.376209 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:10.376181    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d-kube-api-access-5ggk8" pod "24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d" (UID: "24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d"). InnerVolumeSpecName "kube-api-access-5ggk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:10.476367 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:10.476335    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ggk8\" (UniqueName: \"kubernetes.io/projected/24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d-kube-api-access-5ggk8\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:11.058401 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:11.058366    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7288-b102db"]
Oct 30 01:21:11.062967 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:11.062938    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7288-b102db"]
Oct 30 01:21:11.190632 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:11.190594    4864 generic.go:356] "Generic (PLEG): container finished" podID="24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d" containerID="6af92ecfd1b54f99985245c6d593cd129ed236381f54f8159536f82d8a7d0a4b" exitCode=137
Oct 30 01:21:11.190801 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:11.190654    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7288-467c33"
Oct 30 01:21:11.190801 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:11.190691    4864 scope.go:122] "RemoveContainer" containerID="6af92ecfd1b54f99985245c6d593cd129ed236381f54f8159536f82d8a7d0a4b"
Oct 30 01:21:11.201991 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:11.201969    4864 scope.go:122] "RemoveContainer" containerID="6af92ecfd1b54f99985245c6d593cd129ed236381f54f8159536f82d8a7d0a4b"
Oct 30 01:21:11.202355 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:21:11.202334    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6af92ecfd1b54f99985245c6d593cd129ed236381f54f8159536f82d8a7d0a4b\": container with ID starting with 6af92ecfd1b54f99985245c6d593cd129ed236381f54f8159536f82d8a7d0a4b not found: ID does not exist" containerID="6af92ecfd1b54f99985245c6d593cd129ed236381f54f8159536f82d8a7d0a4b"
Oct 30 01:21:11.202416 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:11.202364    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6af92ecfd1b54f99985245c6d593cd129ed236381f54f8159536f82d8a7d0a4b"} err="failed to get container status \"6af92ecfd1b54f99985245c6d593cd129ed236381f54f8159536f82d8a7d0a4b\": rpc error: code = NotFound desc = could not find container \"6af92ecfd1b54f99985245c6d593cd129ed236381f54f8159536f82d8a7d0a4b\": container with ID starting with 6af92ecfd1b54f99985245c6d593cd129ed236381f54f8159536f82d8a7d0a4b not found: ID does not exist"
Oct 30 01:21:11.210666 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:21:11.210638    4864 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7288-467c33\" is forbidden: User \"system:node:ip-10-0-134-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-210.ec2.internal' and this object" podUID="24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d" pod="opendatahub/test-curl-7288-467c33"
Oct 30 01:21:11.217477 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:11.217457    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7288-f3a93a"]
Oct 30 01:21:11.218050 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:11.218038    4864 cpu_manager.go:414] "containerMap: removing container" podUID="cf146b7f-7e1c-497d-9a5d-20bff4bc411e" containerName="test-curl-7288-b102db"
Oct 30 01:21:11.218176 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:11.218052    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf146b7f-7e1c-497d-9a5d-20bff4bc411e" containerName="test-curl-7288-b102db"
Oct 30 01:21:11.218362 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:11.218346    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf146b7f-7e1c-497d-9a5d-20bff4bc411e" containerName="test-curl-7288-b102db"
Oct 30 01:21:11.223120 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:11.223104    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7288-f3a93a"
Oct 30 01:21:11.239291 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:11.239267    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7288-f3a93a"]
Oct 30 01:21:11.268139 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:21:11.268096    4864 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7288-467c33\" is forbidden: User \"system:node:ip-10-0-134-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-210.ec2.internal' and this object" podUID="24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d" pod="opendatahub/test-curl-7288-467c33"
Oct 30 01:21:11.291064 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:11.291034    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t6m9\" (UniqueName: \"kubernetes.io/projected/2b6b915b-9340-4e09-b04e-9b76739acd02-kube-api-access-6t6m9\") pod \"test-curl-7288-f3a93a\" (UID: \"2b6b915b-9340-4e09-b04e-9b76739acd02\") " pod="opendatahub/test-curl-7288-f3a93a"
Oct 30 01:21:11.393309 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:11.393278    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t6m9\" (UniqueName: \"kubernetes.io/projected/2b6b915b-9340-4e09-b04e-9b76739acd02-kube-api-access-6t6m9\") pod \"test-curl-7288-f3a93a\" (UID: \"2b6b915b-9340-4e09-b04e-9b76739acd02\") " pod="opendatahub/test-curl-7288-f3a93a"
Oct 30 01:21:11.414802 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:11.414775    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t6m9\" (UniqueName: \"kubernetes.io/projected/2b6b915b-9340-4e09-b04e-9b76739acd02-kube-api-access-6t6m9\") pod \"test-curl-7288-f3a93a\" (UID: \"2b6b915b-9340-4e09-b04e-9b76739acd02\") " pod="opendatahub/test-curl-7288-f3a93a"
Oct 30 01:21:11.535165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:11.535134    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7288-f3a93a"
Oct 30 01:21:11.794724 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:11.794693    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d" path="/var/lib/kubelet/pods/24ca63b9-fe42-4e3a-a3a1-7c58e9a68c4d/volumes"
Oct 30 01:21:11.830962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:11.830933    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7288-f3a93a"]
Oct 30 01:21:11.832909 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:21:11.832881    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b6b915b_9340_4e09_b04e_9b76739acd02.slice/crio-eb70322598232e924a81f851fb806d9acbd5dc2db906ffadf9833107fa14a32b WatchSource:0}: Error finding container eb70322598232e924a81f851fb806d9acbd5dc2db906ffadf9833107fa14a32b: Status 404 returned error can't find the container with id eb70322598232e924a81f851fb806d9acbd5dc2db906ffadf9833107fa14a32b
Oct 30 01:21:12.199416 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:12.199382    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7288-f3a93a" event={"ID":"2b6b915b-9340-4e09-b04e-9b76739acd02","Type":"ContainerStarted","Data":"04fb6286764059e71c35db2d8c9f67466bb472f43da21849a64dbb22ec2d4339"}
Oct 30 01:21:12.199612 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:12.199588    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7288-f3a93a" event={"ID":"2b6b915b-9340-4e09-b04e-9b76739acd02","Type":"ContainerStarted","Data":"eb70322598232e924a81f851fb806d9acbd5dc2db906ffadf9833107fa14a32b"}
Oct 30 01:21:12.200640 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:12.200590    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7288-b102db" podUID="cf146b7f-7e1c-497d-9a5d-20bff4bc411e" containerName="test-curl-7288-b102db" containerID="cri-o://04ae1674caaad24c3d337ae03c803f4a4752679c238d109e34196bc564c27d2f" gracePeriod=2
Oct 30 01:21:12.235435 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:12.235385    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7288-f3a93a" podStartSLOduration=1.235369918 podStartE2EDuration="1.235369918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:12.232917218 +0000 UTC m=+2384.571019128" watchObservedRunningTime="2026-10-30 01:21:12.235369918 +0000 UTC m=+2384.573471819"
Oct 30 01:21:12.344276 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:12.344249    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7288-7734ce"
Oct 30 01:21:12.406416 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:12.406379    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f19e02b-f300-49b9-9c66-e8e04c3a860b-kube-api-access-2d5v8\" (UniqueName: \"kubernetes.io/projected/7f19e02b-f300-49b9-9c66-e8e04c3a860b-kube-api-access-2d5v8\") pod \"7f19e02b-f300-49b9-9c66-e8e04c3a860b\" (UID: \"7f19e02b-f300-49b9-9c66-e8e04c3a860b\") "
Oct 30 01:21:12.408444 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:12.408422    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f19e02b-f300-49b9-9c66-e8e04c3a860b-kube-api-access-2d5v8" pod "7f19e02b-f300-49b9-9c66-e8e04c3a860b" (UID: "7f19e02b-f300-49b9-9c66-e8e04c3a860b"). InnerVolumeSpecName "kube-api-access-2d5v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:12.507727 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:12.507695    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d5v8\" (UniqueName: \"kubernetes.io/projected/7f19e02b-f300-49b9-9c66-e8e04c3a860b-kube-api-access-2d5v8\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:13.187125 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:13.176161    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7288-f3a93a"]
Oct 30 01:21:13.220093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:13.219185    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7288-f3a93a"]
Oct 30 01:21:13.225582 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:13.224905    4864 generic.go:356] "Generic (PLEG): container finished" podID="7f19e02b-f300-49b9-9c66-e8e04c3a860b" containerID="5f64b76f28fd507a9f0a9d70732660e6ddcd05c1f03e826e5d00518bb441cbab" exitCode=137
Oct 30 01:21:13.225757 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:13.225716    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7288-7734ce"
Oct 30 01:21:13.234597 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:13.233555    4864 scope.go:122] "RemoveContainer" containerID="5f64b76f28fd507a9f0a9d70732660e6ddcd05c1f03e826e5d00518bb441cbab"
Oct 30 01:21:13.252819 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:13.252333    4864 scope.go:122] "RemoveContainer" containerID="5f64b76f28fd507a9f0a9d70732660e6ddcd05c1f03e826e5d00518bb441cbab"
Oct 30 01:21:13.252819 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:21:13.252753    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f64b76f28fd507a9f0a9d70732660e6ddcd05c1f03e826e5d00518bb441cbab\": container with ID starting with 5f64b76f28fd507a9f0a9d70732660e6ddcd05c1f03e826e5d00518bb441cbab not found: ID does not exist" containerID="5f64b76f28fd507a9f0a9d70732660e6ddcd05c1f03e826e5d00518bb441cbab"
Oct 30 01:21:13.252819 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:13.252788    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f64b76f28fd507a9f0a9d70732660e6ddcd05c1f03e826e5d00518bb441cbab"} err="failed to get container status \"5f64b76f28fd507a9f0a9d70732660e6ddcd05c1f03e826e5d00518bb441cbab\": rpc error: code = NotFound desc = could not find container \"5f64b76f28fd507a9f0a9d70732660e6ddcd05c1f03e826e5d00518bb441cbab\": container with ID starting with 5f64b76f28fd507a9f0a9d70732660e6ddcd05c1f03e826e5d00518bb441cbab not found: ID does not exist"
Oct 30 01:21:13.256438 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:21:13.256350    4864 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7288-7734ce\" is forbidden: User \"system:node:ip-10-0-134-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-210.ec2.internal' and this object" podUID="7f19e02b-f300-49b9-9c66-e8e04c3a860b" pod="opendatahub/test-curl-7288-7734ce"
Oct 30 01:21:13.794247 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:13.794217    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f19e02b-f300-49b9-9c66-e8e04c3a860b" path="/var/lib/kubelet/pods/7f19e02b-f300-49b9-9c66-e8e04c3a860b/volumes"
Oct 30 01:21:14.235697 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:14.235669    4864 generic.go:356] "Generic (PLEG): container finished" podID="cf146b7f-7e1c-497d-9a5d-20bff4bc411e" containerID="04ae1674caaad24c3d337ae03c803f4a4752679c238d109e34196bc564c27d2f" exitCode=137
Oct 30 01:21:14.235896 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:14.235881    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7288-f3a93a" podUID="2b6b915b-9340-4e09-b04e-9b76739acd02" containerName="test-curl-7288-f3a93a" containerID="cri-o://04fb6286764059e71c35db2d8c9f67466bb472f43da21849a64dbb22ec2d4339" gracePeriod=2
Oct 30 01:21:14.379339 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:14.379310    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7288-b102db"
Oct 30 01:21:14.435355 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:14.435323    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf146b7f-7e1c-497d-9a5d-20bff4bc411e-kube-api-access-97zs8\" (UniqueName: \"kubernetes.io/projected/cf146b7f-7e1c-497d-9a5d-20bff4bc411e-kube-api-access-97zs8\") pod \"cf146b7f-7e1c-497d-9a5d-20bff4bc411e\" (UID: \"cf146b7f-7e1c-497d-9a5d-20bff4bc411e\") "
Oct 30 01:21:14.437315 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:14.437287    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf146b7f-7e1c-497d-9a5d-20bff4bc411e-kube-api-access-97zs8" pod "cf146b7f-7e1c-497d-9a5d-20bff4bc411e" (UID: "cf146b7f-7e1c-497d-9a5d-20bff4bc411e"). InnerVolumeSpecName "kube-api-access-97zs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:14.536802 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:14.536731    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97zs8\" (UniqueName: \"kubernetes.io/projected/cf146b7f-7e1c-497d-9a5d-20bff4bc411e-kube-api-access-97zs8\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:15.258140 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:15.257238    4864 scope.go:122] "RemoveContainer" containerID="04ae1674caaad24c3d337ae03c803f4a4752679c238d109e34196bc564c27d2f"
Oct 30 01:21:15.258140 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:15.257386    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7288-b102db"
Oct 30 01:21:15.279906 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:21:15.279863    4864 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7288-b102db\" is forbidden: User \"system:node:ip-10-0-134-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-210.ec2.internal' and this object" podUID="cf146b7f-7e1c-497d-9a5d-20bff4bc411e" pod="opendatahub/test-curl-7288-b102db"
Oct 30 01:21:15.794620 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:15.794586    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf146b7f-7e1c-497d-9a5d-20bff4bc411e" path="/var/lib/kubelet/pods/cf146b7f-7e1c-497d-9a5d-20bff4bc411e/volumes"
Oct 30 01:21:16.464274 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:16.464246    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7288-f3a93a"
Oct 30 01:21:16.576983 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:16.576907    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6b915b-9340-4e09-b04e-9b76739acd02-kube-api-access-6t6m9\" (UniqueName: \"kubernetes.io/projected/2b6b915b-9340-4e09-b04e-9b76739acd02-kube-api-access-6t6m9\") pod \"2b6b915b-9340-4e09-b04e-9b76739acd02\" (UID: \"2b6b915b-9340-4e09-b04e-9b76739acd02\") "
Oct 30 01:21:16.579047 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:16.579016    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6b915b-9340-4e09-b04e-9b76739acd02-kube-api-access-6t6m9" pod "2b6b915b-9340-4e09-b04e-9b76739acd02" (UID: "2b6b915b-9340-4e09-b04e-9b76739acd02"). InnerVolumeSpecName "kube-api-access-6t6m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:16.678477 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:16.678448    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t6m9\" (UniqueName: \"kubernetes.io/projected/2b6b915b-9340-4e09-b04e-9b76739acd02-kube-api-access-6t6m9\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:17.271699 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:17.271666    4864 generic.go:356] "Generic (PLEG): container finished" podID="2b6b915b-9340-4e09-b04e-9b76739acd02" containerID="04fb6286764059e71c35db2d8c9f67466bb472f43da21849a64dbb22ec2d4339" exitCode=137
Oct 30 01:21:17.271892 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:17.271734    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7288-f3a93a"
Oct 30 01:21:17.271892 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:17.271753    4864 scope.go:122] "RemoveContainer" containerID="04fb6286764059e71c35db2d8c9f67466bb472f43da21849a64dbb22ec2d4339"
Oct 30 01:21:17.281863 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:17.281676    4864 scope.go:122] "RemoveContainer" containerID="04fb6286764059e71c35db2d8c9f67466bb472f43da21849a64dbb22ec2d4339"
Oct 30 01:21:17.281988 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:21:17.281964    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04fb6286764059e71c35db2d8c9f67466bb472f43da21849a64dbb22ec2d4339\": container with ID starting with 04fb6286764059e71c35db2d8c9f67466bb472f43da21849a64dbb22ec2d4339 not found: ID does not exist" containerID="04fb6286764059e71c35db2d8c9f67466bb472f43da21849a64dbb22ec2d4339"
Oct 30 01:21:17.282128 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:17.281993    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04fb6286764059e71c35db2d8c9f67466bb472f43da21849a64dbb22ec2d4339"} err="failed to get container status \"04fb6286764059e71c35db2d8c9f67466bb472f43da21849a64dbb22ec2d4339\": rpc error: code = NotFound desc = could not find container \"04fb6286764059e71c35db2d8c9f67466bb472f43da21849a64dbb22ec2d4339\": container with ID starting with 04fb6286764059e71c35db2d8c9f67466bb472f43da21849a64dbb22ec2d4339 not found: ID does not exist"
Oct 30 01:21:17.292358 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:21:17.292331    4864 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7288-f3a93a\" is forbidden: User \"system:node:ip-10-0-134-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-210.ec2.internal' and this object" podUID="2b6b915b-9340-4e09-b04e-9b76739acd02" pod="opendatahub/test-curl-7288-f3a93a"
Oct 30 01:21:17.795891 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:17.795475    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b6b915b-9340-4e09-b04e-9b76739acd02" path="/var/lib/kubelet/pods/2b6b915b-9340-4e09-b04e-9b76739acd02/volumes"
Oct 30 01:21:17.800714 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:21:17.800686    4864 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7288-f3a93a\" is forbidden: User \"system:node:ip-10-0-134-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-210.ec2.internal' and this object" podUID="2b6b915b-9340-4e09-b04e-9b76739acd02" pod="opendatahub/test-curl-7288-f3a93a"
Oct 30 01:21:19.039968 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:21:19.039931    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:21:28.134045 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:28.134014    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:21:28.148482 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:28.148454    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:21:28.381375 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:28.381341    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"]
Oct 30 01:21:30.549302 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.549263    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"]
Oct 30 01:21:30.550250 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.550226    4864 cpu_manager.go:414] "containerMap: removing container" podUID="2b6b915b-9340-4e09-b04e-9b76739acd02" containerName="test-curl-7288-f3a93a"
Oct 30 01:21:30.550344 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.550250    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b6b915b-9340-4e09-b04e-9b76739acd02" containerName="test-curl-7288-f3a93a"
Oct 30 01:21:30.550430 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.550417    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b6b915b-9340-4e09-b04e-9b76739acd02" containerName="test-curl-7288-f3a93a"
Oct 30 01:21:30.557569 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.557547    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.588892 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.588854    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-2f820e-openshift-default-dockercfg-qrnt9\""
Oct 30 01:21:30.592965 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.592936    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"]
Oct 30 01:21:30.678688 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.678653    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-token\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.678846 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.678696    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-envoy\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.678956 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.678932    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-credential-socket\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.679043 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.678989    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-podinfo\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.679145 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.679130    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-data\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.679212 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.679197    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-workload-certs\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.679297 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.679284    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-workload-socket\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.679353 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.679316    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istiod-ca-cert\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.679402 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.679360    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mltj6\" (UniqueName: \"kubernetes.io/projected/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-kube-api-access-mltj6\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.780975 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.780928    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-podinfo\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.782226 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.782153    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-data\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.782406 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.782388    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-data\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.788055 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.782912    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-workload-certs\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.788321 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.788300    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-workload-certs\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.788532 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.788516    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-workload-socket\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.789003 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.788977    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istiod-ca-cert\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.798572 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.788910    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-workload-socket\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.798572 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.789758    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istiod-ca-cert\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.800358 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.798872    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mltj6\" (UniqueName: \"kubernetes.io/projected/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-kube-api-access-mltj6\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.800358 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.798953    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-token\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.800358 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.798985    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-envoy\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.800358 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.799167    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-credential-socket\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.800358 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.799546    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-credential-socket\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.807113 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.805380    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-podinfo\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.810036 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.809696    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-envoy\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.821461 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.821431    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mltj6\" (UniqueName: \"kubernetes.io/projected/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-kube-api-access-mltj6\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.823596 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.823568    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-token\") pod \"e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") " pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:30.872058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:30.872027    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:31.207480 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:31.207454    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"]
Oct 30 01:21:31.208965 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:21:31.208942    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd0656_df2b_4cd7_9f6a_ac93a18f9579.slice/crio-12cf0bb74e4a686b0eada052c34d8cedf6cdf4d3a7fe6ab00c2cc29facc71135 WatchSource:0}: Error finding container 12cf0bb74e4a686b0eada052c34d8cedf6cdf4d3a7fe6ab00c2cc29facc71135: Status 404 returned error can't find the container with id 12cf0bb74e4a686b0eada052c34d8cedf6cdf4d3a7fe6ab00c2cc29facc71135
Oct 30 01:21:31.211638 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:31.211609    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:21:31.211707 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:31.211676    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:21:31.211744 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:31.211705    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:21:31.361338 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:31.361287    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8" event={"ID":"41fd0656-df2b-4cd7-9f6a-ac93a18f9579","Type":"ContainerStarted","Data":"40a68cf616bf0a6e16e226ba8b6d9b729a35261b1f4d552168efcb17024f35bd"}
Oct 30 01:21:31.361533 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:31.361374    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8" event={"ID":"41fd0656-df2b-4cd7-9f6a-ac93a18f9579","Type":"ContainerStarted","Data":"12cf0bb74e4a686b0eada052c34d8cedf6cdf4d3a7fe6ab00c2cc29facc71135"}
Oct 30 01:21:31.406019 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:31.405954    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8" podStartSLOduration=1.405936563 podStartE2EDuration="1.405936563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:31.402585065 +0000 UTC m=+2403.740686976" watchObservedRunningTime="2026-10-30 01:21:31.405936563 +0000 UTC m=+2403.744038474"
Oct 30 01:21:31.813036 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:31.813006    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"]
Oct 30 01:21:31.813448 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:31.813206    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t" podUID="c3816261-e881-443f-aa99-889f28c594a5" containerName="istio-proxy" containerID="cri-o://6c3d4acedee66c449d973670c291900fe46eb282609615654f4b7fac8acc02bd" gracePeriod=30
Oct 30 01:21:31.872642 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:31.872606    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:32.879606 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:32.879575    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:33.374131 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:33.374096    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:33.374974 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:33.374952    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:21:37.106820 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.106793    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:21:37.111355 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.111327    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-credential-socket\") pod \"c3816261-e881-443f-aa99-889f28c594a5\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") "
Oct 30 01:21:37.111477 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.111392    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3816261-e881-443f-aa99-889f28c594a5-kube-api-access-xhwzk\" (UniqueName: \"kubernetes.io/projected/c3816261-e881-443f-aa99-889f28c594a5-kube-api-access-xhwzk\") pod \"c3816261-e881-443f-aa99-889f28c594a5\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") "
Oct 30 01:21:37.111477 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.111437    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-istio-data\") pod \"c3816261-e881-443f-aa99-889f28c594a5\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") "
Oct 30 01:21:37.111593 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.111483    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3816261-e881-443f-aa99-889f28c594a5-istio-token\" (UniqueName: \"kubernetes.io/projected/c3816261-e881-443f-aa99-889f28c594a5-istio-token\") pod \"c3816261-e881-443f-aa99-889f28c594a5\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") "
Oct 30 01:21:37.111593 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.111554    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3816261-e881-443f-aa99-889f28c594a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3816261-e881-443f-aa99-889f28c594a5-istio-podinfo\") pod \"c3816261-e881-443f-aa99-889f28c594a5\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") "
Oct 30 01:21:37.111593 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.111561    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-credential-socket" pod "c3816261-e881-443f-aa99-889f28c594a5" (UID: "c3816261-e881-443f-aa99-889f28c594a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:37.111593 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.111586    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-istio-envoy\") pod \"c3816261-e881-443f-aa99-889f28c594a5\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") "
Oct 30 01:21:37.111805 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.111672    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3816261-e881-443f-aa99-889f28c594a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3816261-e881-443f-aa99-889f28c594a5-istiod-ca-cert\") pod \"c3816261-e881-443f-aa99-889f28c594a5\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") "
Oct 30 01:21:37.111805 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.111717    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-istio-data" pod "c3816261-e881-443f-aa99-889f28c594a5" (UID: "c3816261-e881-443f-aa99-889f28c594a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:37.112141 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.112118    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3816261-e881-443f-aa99-889f28c594a5-istiod-ca-cert" pod "c3816261-e881-443f-aa99-889f28c594a5" (UID: "c3816261-e881-443f-aa99-889f28c594a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:37.112476 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.112456    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:37.112532 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.112483    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:37.112532 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.112500    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3816261-e881-443f-aa99-889f28c594a5-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:37.113774 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.113753    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3816261-e881-443f-aa99-889f28c594a5-kube-api-access-xhwzk" pod "c3816261-e881-443f-aa99-889f28c594a5" (UID: "c3816261-e881-443f-aa99-889f28c594a5"). InnerVolumeSpecName "kube-api-access-xhwzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:37.114129 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.114109    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-istio-envoy" pod "c3816261-e881-443f-aa99-889f28c594a5" (UID: "c3816261-e881-443f-aa99-889f28c594a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:37.114222 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.114175    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3816261-e881-443f-aa99-889f28c594a5-istio-podinfo" pod "c3816261-e881-443f-aa99-889f28c594a5" (UID: "c3816261-e881-443f-aa99-889f28c594a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:21:37.114222 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.114178    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3816261-e881-443f-aa99-889f28c594a5-istio-token" pod "c3816261-e881-443f-aa99-889f28c594a5" (UID: "c3816261-e881-443f-aa99-889f28c594a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:37.213112 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.213039    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-workload-socket\") pod \"c3816261-e881-443f-aa99-889f28c594a5\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") "
Oct 30 01:21:37.213312 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.213121    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-workload-certs\") pod \"c3816261-e881-443f-aa99-889f28c594a5\" (UID: \"c3816261-e881-443f-aa99-889f28c594a5\") "
Oct 30 01:21:37.213376 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.213330    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-workload-socket" pod "c3816261-e881-443f-aa99-889f28c594a5" (UID: "c3816261-e881-443f-aa99-889f28c594a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:37.213428 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.213405    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-workload-certs" pod "c3816261-e881-443f-aa99-889f28c594a5" (UID: "c3816261-e881-443f-aa99-889f28c594a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:37.213784 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.213766    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhwzk\" (UniqueName: \"kubernetes.io/projected/c3816261-e881-443f-aa99-889f28c594a5-kube-api-access-xhwzk\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:37.213829 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.213791    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3816261-e881-443f-aa99-889f28c594a5-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:37.213829 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.213806    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3816261-e881-443f-aa99-889f28c594a5-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:37.213829 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.213818    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:37.213921 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.213831    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:37.213921 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.213844    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3816261-e881-443f-aa99-889f28c594a5-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:37.498920 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.498882    4864 generic.go:356] "Generic (PLEG): container finished" podID="c3816261-e881-443f-aa99-889f28c594a5" containerID="6c3d4acedee66c449d973670c291900fe46eb282609615654f4b7fac8acc02bd" exitCode=0
Oct 30 01:21:37.499256 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.499151    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t" event={"ID":"c3816261-e881-443f-aa99-889f28c594a5","Type":"ContainerDied","Data":"6c3d4acedee66c449d973670c291900fe46eb282609615654f4b7fac8acc02bd"}
Oct 30 01:21:37.499256 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.499208    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t" event={"ID":"c3816261-e881-443f-aa99-889f28c594a5","Type":"ContainerDied","Data":"85a06657526e8e354bf16b8f02420670c74c3b21eadf71e022d633bb5aeab0bc"}
Oct 30 01:21:37.499256 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.499239    4864 scope.go:122] "RemoveContainer" containerID="6c3d4acedee66c449d973670c291900fe46eb282609615654f4b7fac8acc02bd"
Oct 30 01:21:37.499485 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.499425    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"
Oct 30 01:21:37.517337 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.517302    4864 scope.go:122] "RemoveContainer" containerID="6c3d4acedee66c449d973670c291900fe46eb282609615654f4b7fac8acc02bd"
Oct 30 01:21:37.518434 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:21:37.518407    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c3d4acedee66c449d973670c291900fe46eb282609615654f4b7fac8acc02bd\": container with ID starting with 6c3d4acedee66c449d973670c291900fe46eb282609615654f4b7fac8acc02bd not found: ID does not exist" containerID="6c3d4acedee66c449d973670c291900fe46eb282609615654f4b7fac8acc02bd"
Oct 30 01:21:37.518542 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.518444    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3d4acedee66c449d973670c291900fe46eb282609615654f4b7fac8acc02bd"} err="failed to get container status \"6c3d4acedee66c449d973670c291900fe46eb282609615654f4b7fac8acc02bd\": rpc error: code = NotFound desc = could not find container \"6c3d4acedee66c449d973670c291900fe46eb282609615654f4b7fac8acc02bd\": container with ID starting with 6c3d4acedee66c449d973670c291900fe46eb282609615654f4b7fac8acc02bd not found: ID does not exist"
Oct 30 01:21:37.558044 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.558007    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"]
Oct 30 01:21:37.563001 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.562973    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-fgh5t"]
Oct 30 01:21:37.800536 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:37.800459    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3816261-e881-443f-aa99-889f28c594a5" path="/var/lib/kubelet/pods/c3816261-e881-443f-aa99-889f28c594a5/volumes"
Oct 30 01:21:41.038643 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:41.037642    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"]
Oct 30 01:21:41.038643 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:41.037885    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p" podUID="fb21179a-656e-4c81-8c3e-fa670ec27c22" containerName="istio-proxy" containerID="cri-o://ce3532a6c368b5327eefd8eb672b4aec94b409a46cfb0d36b1c1836dd42d4805" gracePeriod=30
Oct 30 01:21:43.412325 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:43.412279    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r" podUID="ed407c49-7ffe-47bc-a5aa-e4e336f92130" containerName="main" containerID="cri-o://87f3e2d11ad054972be57e634fdba51f9c85275a0eb062764ada1044569c3ead" gracePeriod=45
Oct 30 01:21:43.544033 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:43.544000    4864 generic.go:356] "Generic (PLEG): container finished" podID="ed407c49-7ffe-47bc-a5aa-e4e336f92130" containerID="87f3e2d11ad054972be57e634fdba51f9c85275a0eb062764ada1044569c3ead" exitCode=0
Oct 30 01:21:43.544205 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:43.544124    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r" event={"ID":"ed407c49-7ffe-47bc-a5aa-e4e336f92130","Type":"ContainerDied","Data":"87f3e2d11ad054972be57e634fdba51f9c85275a0eb062764ada1044569c3ead"}
Oct 30 01:21:43.699556 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:43.699524    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:21:43.736333 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:43.736304    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-home\" (UniqueName: \"kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-home\") pod \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\" (UID: \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\") "
Oct 30 01:21:43.736500 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:43.736382    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-dshm\") pod \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\" (UID: \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\") "
Oct 30 01:21:43.736500 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:43.736411    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-tmp-dir\") pod \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\" (UID: \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\") "
Oct 30 01:21:43.736500 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:43.736492    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-model-cache\") pod \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\" (UID: \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\") "
Oct 30 01:21:43.736675 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:43.736560    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed407c49-7ffe-47bc-a5aa-e4e336f92130-kube-api-access-wq946\" (UniqueName: \"kubernetes.io/projected/ed407c49-7ffe-47bc-a5aa-e4e336f92130-kube-api-access-wq946\") pod \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\" (UID: \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\") "
Oct 30 01:21:43.736675 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:43.736605    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-home" pod "ed407c49-7ffe-47bc-a5aa-e4e336f92130" (UID: "ed407c49-7ffe-47bc-a5aa-e4e336f92130"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:43.736675 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:43.736645    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed407c49-7ffe-47bc-a5aa-e4e336f92130-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed407c49-7ffe-47bc-a5aa-e4e336f92130-tls-certs\") pod \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\" (UID: \"ed407c49-7ffe-47bc-a5aa-e4e336f92130\") "
Oct 30 01:21:43.736807 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:43.736750    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-tmp-dir" pod "ed407c49-7ffe-47bc-a5aa-e4e336f92130" (UID: "ed407c49-7ffe-47bc-a5aa-e4e336f92130"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:43.736849 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:43.736822    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-model-cache" pod "ed407c49-7ffe-47bc-a5aa-e4e336f92130" (UID: "ed407c49-7ffe-47bc-a5aa-e4e336f92130"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:43.737291 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:43.737239    4864 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-model-cache\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:43.737291 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:43.737262    4864 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-home\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:43.737291 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:43.737273    4864 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-tmp-dir\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:43.739349 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:43.739304    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed407c49-7ffe-47bc-a5aa-e4e336f92130-tls-certs" pod "ed407c49-7ffe-47bc-a5aa-e4e336f92130" (UID: "ed407c49-7ffe-47bc-a5aa-e4e336f92130"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:21:43.739517 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:43.739496    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed407c49-7ffe-47bc-a5aa-e4e336f92130-kube-api-access-wq946" pod "ed407c49-7ffe-47bc-a5aa-e4e336f92130" (UID: "ed407c49-7ffe-47bc-a5aa-e4e336f92130"). InnerVolumeSpecName "kube-api-access-wq946". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:43.739609 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:43.739590    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-dshm" pod "ed407c49-7ffe-47bc-a5aa-e4e336f92130" (UID: "ed407c49-7ffe-47bc-a5aa-e4e336f92130"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:43.838608 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:43.838571    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq946\" (UniqueName: \"kubernetes.io/projected/ed407c49-7ffe-47bc-a5aa-e4e336f92130-kube-api-access-wq946\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:43.838783 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:43.838614    4864 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed407c49-7ffe-47bc-a5aa-e4e336f92130-tls-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:43.838783 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:43.838631    4864 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed407c49-7ffe-47bc-a5aa-e4e336f92130-dshm\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:44.551198 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:44.551170    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"
Oct 30 01:21:44.551636 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:44.551212    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r" event={"ID":"ed407c49-7ffe-47bc-a5aa-e4e336f92130","Type":"ContainerDied","Data":"30c69be82ad63030037cfafc507c50a168b7e13ced83b341841bbffc243e744e"}
Oct 30 01:21:44.551636 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:44.551267    4864 scope.go:122] "RemoveContainer" containerID="87f3e2d11ad054972be57e634fdba51f9c85275a0eb062764ada1044569c3ead"
Oct 30 01:21:44.600717 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:44.600685    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"]
Oct 30 01:21:44.608435 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:44.608407    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c666c2/auth-test-model-c666c2-kserve-78cd5cf76f-2292r"]
Oct 30 01:21:45.794221 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:45.794175    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed407c49-7ffe-47bc-a5aa-e4e336f92130" path="/var/lib/kubelet/pods/ed407c49-7ffe-47bc-a5aa-e4e336f92130/volumes"
Oct 30 01:21:46.565331 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.565300    4864 generic.go:356] "Generic (PLEG): container finished" podID="fb21179a-656e-4c81-8c3e-fa670ec27c22" containerID="ce3532a6c368b5327eefd8eb672b4aec94b409a46cfb0d36b1c1836dd42d4805" exitCode=0
Oct 30 01:21:46.565462 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.565368    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p" event={"ID":"fb21179a-656e-4c81-8c3e-fa670ec27c22","Type":"ContainerDied","Data":"ce3532a6c368b5327eefd8eb672b4aec94b409a46cfb0d36b1c1836dd42d4805"}
Oct 30 01:21:46.710144 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.710119    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:21:46.777633 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.777595    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-workload-certs\") pod \"fb21179a-656e-4c81-8c3e-fa670ec27c22\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") "
Oct 30 01:21:46.777822 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.777657    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-credential-socket\") pod \"fb21179a-656e-4c81-8c3e-fa670ec27c22\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") "
Oct 30 01:21:46.777822 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.777759    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-envoy\") pod \"fb21179a-656e-4c81-8c3e-fa670ec27c22\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") "
Oct 30 01:21:46.777822 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.777806    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-podinfo\") pod \"fb21179a-656e-4c81-8c3e-fa670ec27c22\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") "
Oct 30 01:21:46.778005 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.777872    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-token\" (UniqueName: \"kubernetes.io/projected/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-token\") pod \"fb21179a-656e-4c81-8c3e-fa670ec27c22\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") "
Oct 30 01:21:46.778005 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.777922    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb21179a-656e-4c81-8c3e-fa670ec27c22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb21179a-656e-4c81-8c3e-fa670ec27c22-istiod-ca-cert\") pod \"fb21179a-656e-4c81-8c3e-fa670ec27c22\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") "
Oct 30 01:21:46.778005 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.777921    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-workload-certs" pod "fb21179a-656e-4c81-8c3e-fa670ec27c22" (UID: "fb21179a-656e-4c81-8c3e-fa670ec27c22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:46.778005 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.777974    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-workload-socket\") pod \"fb21179a-656e-4c81-8c3e-fa670ec27c22\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") "
Oct 30 01:21:46.778247 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.777998    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-credential-socket" pod "fb21179a-656e-4c81-8c3e-fa670ec27c22" (UID: "fb21179a-656e-4c81-8c3e-fa670ec27c22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:46.778247 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.778014    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb21179a-656e-4c81-8c3e-fa670ec27c22-kube-api-access-q22vs\" (UniqueName: \"kubernetes.io/projected/fb21179a-656e-4c81-8c3e-fa670ec27c22-kube-api-access-q22vs\") pod \"fb21179a-656e-4c81-8c3e-fa670ec27c22\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") "
Oct 30 01:21:46.778247 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.778104    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-data\") pod \"fb21179a-656e-4c81-8c3e-fa670ec27c22\" (UID: \"fb21179a-656e-4c81-8c3e-fa670ec27c22\") "
Oct 30 01:21:46.778741 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.778648    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:46.778741 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.778674    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:46.779178 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.778985    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb21179a-656e-4c81-8c3e-fa670ec27c22-istiod-ca-cert" pod "fb21179a-656e-4c81-8c3e-fa670ec27c22" (UID: "fb21179a-656e-4c81-8c3e-fa670ec27c22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:46.779178 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.779104    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-workload-socket" pod "fb21179a-656e-4c81-8c3e-fa670ec27c22" (UID: "fb21179a-656e-4c81-8c3e-fa670ec27c22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:46.779714 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.779673    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-data" pod "fb21179a-656e-4c81-8c3e-fa670ec27c22" (UID: "fb21179a-656e-4c81-8c3e-fa670ec27c22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:46.780897 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.780557    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-envoy" pod "fb21179a-656e-4c81-8c3e-fa670ec27c22" (UID: "fb21179a-656e-4c81-8c3e-fa670ec27c22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:46.780897 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.780606    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-token" pod "fb21179a-656e-4c81-8c3e-fa670ec27c22" (UID: "fb21179a-656e-4c81-8c3e-fa670ec27c22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:46.780897 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.780698    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-podinfo" pod "fb21179a-656e-4c81-8c3e-fa670ec27c22" (UID: "fb21179a-656e-4c81-8c3e-fa670ec27c22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:21:46.781337 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.781308    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb21179a-656e-4c81-8c3e-fa670ec27c22-kube-api-access-q22vs" pod "fb21179a-656e-4c81-8c3e-fa670ec27c22" (UID: "fb21179a-656e-4c81-8c3e-fa670ec27c22"). InnerVolumeSpecName "kube-api-access-q22vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:46.880266 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.880187    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:46.880266 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.880223    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q22vs\" (UniqueName: \"kubernetes.io/projected/fb21179a-656e-4c81-8c3e-fa670ec27c22-kube-api-access-q22vs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:46.880266 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.880234    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:46.880266 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.880243    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:46.880266 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.880251    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:46.880266 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.880259    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb21179a-656e-4c81-8c3e-fa670ec27c22-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:46.880266 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:46.880268    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb21179a-656e-4c81-8c3e-fa670ec27c22-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:21:47.574082 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:47.574049    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"
Oct 30 01:21:47.574283 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:47.574050    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p" event={"ID":"fb21179a-656e-4c81-8c3e-fa670ec27c22","Type":"ContainerDied","Data":"e27d6a1924406dc07dd603e00834263f2f56df645b2a450f440abd4fed64fc4d"}
Oct 30 01:21:47.574283 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:47.574228    4864 scope.go:122] "RemoveContainer" containerID="ce3532a6c368b5327eefd8eb672b4aec94b409a46cfb0d36b1c1836dd42d4805"
Oct 30 01:21:47.648541 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:47.648503    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"]
Oct 30 01:21:47.658260 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:47.658227    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-zr22p"]
Oct 30 01:21:47.793430 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:47.793390    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb21179a-656e-4c81-8c3e-fa670ec27c22" path="/var/lib/kubelet/pods/fb21179a-656e-4c81-8c3e-fa670ec27c22/volumes"
Oct 30 01:21:49.041994 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:21:49.041958    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:21:50.664940 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:50.664909    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"]
Oct 30 01:21:50.665521 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:50.665505    4864 cpu_manager.go:414] "containerMap: removing container" podUID="ed407c49-7ffe-47bc-a5aa-e4e336f92130" containerName="main"
Oct 30 01:21:50.665567 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:50.665522    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed407c49-7ffe-47bc-a5aa-e4e336f92130" containerName="main"
Oct 30 01:21:50.665567 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:50.665553    4864 cpu_manager.go:414] "containerMap: removing container" podUID="c3816261-e881-443f-aa99-889f28c594a5" containerName="istio-proxy"
Oct 30 01:21:50.665567 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:50.665559    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3816261-e881-443f-aa99-889f28c594a5" containerName="istio-proxy"
Oct 30 01:21:50.665665 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:50.665570    4864 cpu_manager.go:414] "containerMap: removing container" podUID="fb21179a-656e-4c81-8c3e-fa670ec27c22" containerName="istio-proxy"
Oct 30 01:21:50.665665 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:50.665575    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb21179a-656e-4c81-8c3e-fa670ec27c22" containerName="istio-proxy"
Oct 30 01:21:50.665665 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:50.665658    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed407c49-7ffe-47bc-a5aa-e4e336f92130" containerName="main"
Oct 30 01:21:50.665752 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:50.665671    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3816261-e881-443f-aa99-889f28c594a5" containerName="istio-proxy"
Oct 30 01:21:50.665752 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:50.665679    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb21179a-656e-4c81-8c3e-fa670ec27c22" containerName="istio-proxy"
Oct 30 01:21:50.670389 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:50.670371    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"
Oct 30 01:21:50.694133 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:50.694102    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-499482-serving-cert\""
Oct 30 01:21:50.698649 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:50.698616    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"]
Oct 30 01:21:50.729144 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:50.729113    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac76439d-8c30-4810-b242-1a3e1ed2b556-maas-api-tls\") pod \"maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp\" (UID: \"ac76439d-8c30-4810-b242-1a3e1ed2b556\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"
Oct 30 01:21:50.729336 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:50.729194    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hml27\" (UniqueName: \"kubernetes.io/projected/ac76439d-8c30-4810-b242-1a3e1ed2b556-kube-api-access-hml27\") pod \"maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp\" (UID: \"ac76439d-8c30-4810-b242-1a3e1ed2b556\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"
Oct 30 01:21:50.729336 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:50.729254    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac76439d-8c30-4810-b242-1a3e1ed2b556-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp\" (UID: \"ac76439d-8c30-4810-b242-1a3e1ed2b556\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"
Oct 30 01:21:50.729462 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:50.729369    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac76439d-8c30-4810-b242-1a3e1ed2b556-metrics-certs\") pod \"maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp\" (UID: \"ac76439d-8c30-4810-b242-1a3e1ed2b556\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"
Oct 30 01:21:50.831135 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:50.831064    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hml27\" (UniqueName: \"kubernetes.io/projected/ac76439d-8c30-4810-b242-1a3e1ed2b556-kube-api-access-hml27\") pod \"maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp\" (UID: \"ac76439d-8c30-4810-b242-1a3e1ed2b556\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"
Oct 30 01:21:50.831313 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:50.831175    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac76439d-8c30-4810-b242-1a3e1ed2b556-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp\" (UID: \"ac76439d-8c30-4810-b242-1a3e1ed2b556\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"
Oct 30 01:21:50.831355 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:50.831312    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac76439d-8c30-4810-b242-1a3e1ed2b556-metrics-certs\") pod \"maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp\" (UID: \"ac76439d-8c30-4810-b242-1a3e1ed2b556\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"
Oct 30 01:21:50.831472 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:50.831453    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac76439d-8c30-4810-b242-1a3e1ed2b556-maas-api-tls\") pod \"maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp\" (UID: \"ac76439d-8c30-4810-b242-1a3e1ed2b556\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"
Oct 30 01:21:50.831565 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:21:50.831543    4864 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-serving-cert: secret "maas-api-e2e-extmod-a-499482-serving-cert" not found
Oct 30 01:21:50.831790 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:21:50.831771    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac76439d-8c30-4810-b242-1a3e1ed2b556-maas-api-tls podName:ac76439d-8c30-4810-b242-1a3e1ed2b556 nodeName:}" failed. No retries permitted until 2026-10-30 01:21:51.331752201 +0000 UTC m=+2423.669854096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac76439d-8c30-4810-b242-1a3e1ed2b556-maas-api-tls") pod "maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp" (UID: "ac76439d-8c30-4810-b242-1a3e1ed2b556") : secret "maas-api-e2e-extmod-a-499482-serving-cert" not found
Oct 30 01:21:50.832096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:50.832056    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac76439d-8c30-4810-b242-1a3e1ed2b556-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp\" (UID: \"ac76439d-8c30-4810-b242-1a3e1ed2b556\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"
Oct 30 01:21:50.833855 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:50.833837    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac76439d-8c30-4810-b242-1a3e1ed2b556-metrics-certs\") pod \"maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp\" (UID: \"ac76439d-8c30-4810-b242-1a3e1ed2b556\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"
Oct 30 01:21:50.853770 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:50.853737    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hml27\" (UniqueName: \"kubernetes.io/projected/ac76439d-8c30-4810-b242-1a3e1ed2b556-kube-api-access-hml27\") pod \"maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp\" (UID: \"ac76439d-8c30-4810-b242-1a3e1ed2b556\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"
Oct 30 01:21:51.337519 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:51.337473    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac76439d-8c30-4810-b242-1a3e1ed2b556-maas-api-tls\") pod \"maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp\" (UID: \"ac76439d-8c30-4810-b242-1a3e1ed2b556\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"
Oct 30 01:21:51.339935 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:51.339904    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac76439d-8c30-4810-b242-1a3e1ed2b556-maas-api-tls\") pod \"maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp\" (UID: \"ac76439d-8c30-4810-b242-1a3e1ed2b556\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"
Oct 30 01:21:51.584380 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:51.584351    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"
Oct 30 01:21:51.914129 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:51.914033    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"]
Oct 30 01:21:51.959804 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:51.959774    4864 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:21:52.611548 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:52.611453    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp" event={"ID":"ac76439d-8c30-4810-b242-1a3e1ed2b556","Type":"ContainerStarted","Data":"21a8e27de431d5693ff1ec193b81ac9ec47e372cad70a9f3d354206048e8d87f"}
Oct 30 01:21:52.611695 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:52.611554    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"
Oct 30 01:21:52.611695 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:52.611583    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp" event={"ID":"ac76439d-8c30-4810-b242-1a3e1ed2b556","Type":"ContainerStarted","Data":"9a08d0667ac5837f4dc6116f2186ea3f9f887c27c072aed5719e7580b0abfd10"}
Oct 30 01:21:52.651531 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:52.651468    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp" podStartSLOduration=2.309760989 podStartE2EDuration="2.651448956s" totalImagesPullingTime="341.687967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:52.650994096 +0000 UTC m=+2424.989096009" watchObservedRunningTime="2026-10-30 01:21:52.651448956 +0000 UTC m=+2424.989550866"
Oct 30 01:21:58.072678 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.072639    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"]
Oct 30 01:21:58.077538 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.077520    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.091104 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.091067    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c7bba9-openshift-default-dockercfg-j4g96\""
Oct 30 01:21:58.107083 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.107032    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"]
Oct 30 01:21:58.193094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.176888    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-token\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.193094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.176995    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-workload-socket\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.193094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.177043    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istiod-ca-cert\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.193094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.177116    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-data\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.193094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.177154    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-envoy\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.193094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.177249    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8w9t\" (UniqueName: \"kubernetes.io/projected/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-kube-api-access-h8w9t\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.193094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.177284    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-workload-certs\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.193094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.177373    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-podinfo\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.193094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.177423    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-credential-socket\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.278911 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.278875    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-workload-socket\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.278911 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.278926    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istiod-ca-cert\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.279196 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.278959    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-data\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.279196 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.278980    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-envoy\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.279196 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.279059    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8w9t\" (UniqueName: \"kubernetes.io/projected/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-kube-api-access-h8w9t\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.279196 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.279132    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-workload-certs\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.279427 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.279221    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-podinfo\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.279427 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.279295    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-credential-socket\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.279427 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.279339    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-token\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.279427 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.279390    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-workload-socket\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.279604 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.279431    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-data\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.279833 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.279803    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istiod-ca-cert\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.279939 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.279895    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-workload-certs\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.280004 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.279980    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-credential-socket\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.281978 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.281950    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-envoy\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.282244 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.282226    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-podinfo\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.301453 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.301417    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-token\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.301775 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.301748    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8w9t\" (UniqueName: \"kubernetes.io/projected/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-kube-api-access-h8w9t\") pod \"e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") " pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.391817 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.391721    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:21:58.623313 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.623284    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"
Oct 30 01:21:58.716569 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:21:58.716534    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf8fc6a_2b8a_420e_83e1_b5ba04312b9c.slice/crio-a3764851c176c9358216103842aae91fe5a397c85203948fe90eafcc86a9167b WatchSource:0}: Error finding container a3764851c176c9358216103842aae91fe5a397c85203948fe90eafcc86a9167b: Status 404 returned error can't find the container with id a3764851c176c9358216103842aae91fe5a397c85203948fe90eafcc86a9167b
Oct 30 01:21:58.716904 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.716878    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"]
Oct 30 01:21:58.719814 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.719779    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:21:58.719921 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.719849    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:21:58.719921 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:58.719878    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:21:59.656271 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:59.656236    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4" event={"ID":"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c","Type":"ContainerStarted","Data":"34f56cf60c8df35eae8740bda92891c5f88f45be1bb06a55c2b0ee36f44aa07f"}
Oct 30 01:21:59.656754 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:59.656286    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4" event={"ID":"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c","Type":"ContainerStarted","Data":"a3764851c176c9358216103842aae91fe5a397c85203948fe90eafcc86a9167b"}
Oct 30 01:21:59.714821 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:21:59.714766    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4" podStartSLOduration=1.7147514419999998 podStartE2EDuration="1.714751442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:59.711087751 +0000 UTC m=+2432.049189658" watchObservedRunningTime="2026-10-30 01:21:59.714751442 +0000 UTC m=+2432.052853352"
Oct 30 01:22:00.392130 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:00.392094    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:22:00.397414 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:00.397388    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:22:00.662406 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:00.662318    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:22:00.663208 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:00.663188    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:22:09.341433 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.341397    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"]
Oct 30 01:22:09.346979 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.346954    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"
Oct 30 01:22:09.362563 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.362536    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-a47b9a-serving-cert\""
Oct 30 01:22:09.369762 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.369728    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2752235c-68d2-4be4-b1eb-d851d2b5c903-metrics-certs\") pod \"maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf\" (UID: \"2752235c-68d2-4be4-b1eb-d851d2b5c903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"
Oct 30 01:22:09.369951 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.369805    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzhrq\" (UniqueName: \"kubernetes.io/projected/2752235c-68d2-4be4-b1eb-d851d2b5c903-kube-api-access-hzhrq\") pod \"maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf\" (UID: \"2752235c-68d2-4be4-b1eb-d851d2b5c903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"
Oct 30 01:22:09.369951 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.369890    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2752235c-68d2-4be4-b1eb-d851d2b5c903-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf\" (UID: \"2752235c-68d2-4be4-b1eb-d851d2b5c903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"
Oct 30 01:22:09.369951 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.369923    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2752235c-68d2-4be4-b1eb-d851d2b5c903-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf\" (UID: \"2752235c-68d2-4be4-b1eb-d851d2b5c903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"
Oct 30 01:22:09.377474 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.377442    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"]
Oct 30 01:22:09.471212 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.471177    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2752235c-68d2-4be4-b1eb-d851d2b5c903-metrics-certs\") pod \"maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf\" (UID: \"2752235c-68d2-4be4-b1eb-d851d2b5c903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"
Oct 30 01:22:09.471413 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.471243    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzhrq\" (UniqueName: \"kubernetes.io/projected/2752235c-68d2-4be4-b1eb-d851d2b5c903-kube-api-access-hzhrq\") pod \"maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf\" (UID: \"2752235c-68d2-4be4-b1eb-d851d2b5c903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"
Oct 30 01:22:09.471413 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.471316    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2752235c-68d2-4be4-b1eb-d851d2b5c903-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf\" (UID: \"2752235c-68d2-4be4-b1eb-d851d2b5c903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"
Oct 30 01:22:09.471413 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.471351    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2752235c-68d2-4be4-b1eb-d851d2b5c903-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf\" (UID: \"2752235c-68d2-4be4-b1eb-d851d2b5c903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"
Oct 30 01:22:09.471606 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:22:09.471461    4864 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-serving-cert: secret "maas-api-e2e-extmod-a-a47b9a-serving-cert" not found
Oct 30 01:22:09.471606 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:22:09.471556    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2752235c-68d2-4be4-b1eb-d851d2b5c903-maas-api-tls podName:2752235c-68d2-4be4-b1eb-d851d2b5c903 nodeName:}" failed. No retries permitted until 2026-10-30 01:22:09.97153597 +0000 UTC m=+2442.309637865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2752235c-68d2-4be4-b1eb-d851d2b5c903-maas-api-tls") pod "maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf" (UID: "2752235c-68d2-4be4-b1eb-d851d2b5c903") : secret "maas-api-e2e-extmod-a-a47b9a-serving-cert" not found
Oct 30 01:22:09.472033 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.472014    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2752235c-68d2-4be4-b1eb-d851d2b5c903-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf\" (UID: \"2752235c-68d2-4be4-b1eb-d851d2b5c903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"
Oct 30 01:22:09.473818 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.473795    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2752235c-68d2-4be4-b1eb-d851d2b5c903-metrics-certs\") pod \"maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf\" (UID: \"2752235c-68d2-4be4-b1eb-d851d2b5c903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"
Oct 30 01:22:09.495187 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.495166    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzhrq\" (UniqueName: \"kubernetes.io/projected/2752235c-68d2-4be4-b1eb-d851d2b5c903-kube-api-access-hzhrq\") pod \"maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf\" (UID: \"2752235c-68d2-4be4-b1eb-d851d2b5c903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"
Oct 30 01:22:09.742951 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.742916    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"]
Oct 30 01:22:09.747309 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.747291    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.771213 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.771055    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-9da49f-openshift-default-dockercfg-9sqgl\""
Oct 30 01:22:09.774484 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.774457    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-envoy\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.774616 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.774564    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-workload-certs\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.774616 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.774595    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-workload-socket\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.774726 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.774628    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-token\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.774726 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.774644    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istiod-ca-cert\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.774726 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.774691    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"]
Oct 30 01:22:09.774869 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.774699    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xlv2\" (UniqueName: \"kubernetes.io/projected/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-kube-api-access-2xlv2\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.774869 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.774805    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-credential-socket\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.774869 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.774825    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-podinfo\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.774869 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.774842    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-data\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.875551 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.875515    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-credential-socket\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.875551 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.875552    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-podinfo\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.875754 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.875571    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-data\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.875754 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.875623    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-envoy\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.875754 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.875708    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-workload-certs\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.875754 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.875736    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-workload-socket\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.875941 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.875777    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-token\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.875941 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.875801    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istiod-ca-cert\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.875941 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.875870    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xlv2\" (UniqueName: \"kubernetes.io/projected/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-kube-api-access-2xlv2\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.876086 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.875949    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-credential-socket\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.876086 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.875958    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-data\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.876331 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.876305    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-workload-socket\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.876412 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.876346    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-workload-certs\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.876531 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.876512    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istiod-ca-cert\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.877970 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.877946    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-envoy\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.878222 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.878201    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-podinfo\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.897285 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.897261    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-token\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.897383 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.897344    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xlv2\" (UniqueName: \"kubernetes.io/projected/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-kube-api-access-2xlv2\") pod \"e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") " pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:09.976902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.976867    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2752235c-68d2-4be4-b1eb-d851d2b5c903-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf\" (UID: \"2752235c-68d2-4be4-b1eb-d851d2b5c903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"
Oct 30 01:22:09.979272 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:09.979246    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2752235c-68d2-4be4-b1eb-d851d2b5c903-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf\" (UID: \"2752235c-68d2-4be4-b1eb-d851d2b5c903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"
Oct 30 01:22:10.061172 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:10.061065    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:10.261183 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:10.261153    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"
Oct 30 01:22:10.382200 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:10.382159    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"]
Oct 30 01:22:10.382720 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:22:10.382283    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa394cb_eba6_4e15_9a0e_523e2379f7ac.slice/crio-6b0d4712ab3d91f752cb3ca6599d1932abb678d1b77936db5f62b9092577d910 WatchSource:0}: Error finding container 6b0d4712ab3d91f752cb3ca6599d1932abb678d1b77936db5f62b9092577d910: Status 404 returned error can't find the container with id 6b0d4712ab3d91f752cb3ca6599d1932abb678d1b77936db5f62b9092577d910
Oct 30 01:22:10.385559 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:10.385527    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:22:10.385641 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:10.385589    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:22:10.385641 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:10.385618    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:22:10.575389 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:10.575307    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"]
Oct 30 01:22:10.575848 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:22:10.575822    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2752235c_68d2_4be4_b1eb_d851d2b5c903.slice/crio-79fa677f24f978855379a1ea8058abeb5af843e168221f751e99a4e06c6950ce WatchSource:0}: Error finding container 79fa677f24f978855379a1ea8058abeb5af843e168221f751e99a4e06c6950ce: Status 404 returned error can't find the container with id 79fa677f24f978855379a1ea8058abeb5af843e168221f751e99a4e06c6950ce
Oct 30 01:22:10.728801 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:10.728763    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf" event={"ID":"2752235c-68d2-4be4-b1eb-d851d2b5c903","Type":"ContainerStarted","Data":"79fa677f24f978855379a1ea8058abeb5af843e168221f751e99a4e06c6950ce"}
Oct 30 01:22:10.730609 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:10.730577    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z" event={"ID":"3aa394cb-eba6-4e15-9a0e-523e2379f7ac","Type":"ContainerStarted","Data":"ed5252895c5d24c30c3931649ccc0b63aa0f09fff0f0551fbe1a03daa6f62bd0"}
Oct 30 01:22:10.730749 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:10.730630    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z" event={"ID":"3aa394cb-eba6-4e15-9a0e-523e2379f7ac","Type":"ContainerStarted","Data":"6b0d4712ab3d91f752cb3ca6599d1932abb678d1b77936db5f62b9092577d910"}
Oct 30 01:22:10.784959 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:10.784897    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z" podStartSLOduration=1.7848838360000001 podStartE2EDuration="1.784883836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:10.782843354 +0000 UTC m=+2443.120945269" watchObservedRunningTime="2026-10-30 01:22:10.784883836 +0000 UTC m=+2443.122985745"
Oct 30 01:22:11.067095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:11.062381    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:11.084279 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:11.084222    4864 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 01:22:11.084460 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:11.084303    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z" podUID="3aa394cb-eba6-4e15-9a0e-523e2379f7ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 01:22:11.738340 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:11.738301    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf" event={"ID":"2752235c-68d2-4be4-b1eb-d851d2b5c903","Type":"ContainerStarted","Data":"111a6509afe8eed4eb8a2bf83776bbf3bb681e8ccba1ab05196b097e9a90faa2"}
Oct 30 01:22:11.738735 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:11.738505    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"
Oct 30 01:22:11.784002 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:11.783934    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf" podStartSLOduration=2.458003024 podStartE2EDuration="2.783918323s" totalImagesPullingTime="325.915299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:11.781608815 +0000 UTC m=+2444.119710726" watchObservedRunningTime="2026-10-30 01:22:11.783918323 +0000 UTC m=+2444.122020224"
Oct 30 01:22:12.080153 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:12.077175    4864 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 01:22:12.080153 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:12.077253    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z" podUID="3aa394cb-eba6-4e15-9a0e-523e2379f7ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 01:22:13.070905 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:13.070463    4864 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 01:22:13.070905 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:13.070526    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z" podUID="3aa394cb-eba6-4e15-9a0e-523e2379f7ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 01:22:14.080096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:14.077658    4864 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 01:22:14.080096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:14.077732    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z" podUID="3aa394cb-eba6-4e15-9a0e-523e2379f7ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 01:22:15.065093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:15.062335    4864 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 01:22:15.065093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:15.062393    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z" podUID="3aa394cb-eba6-4e15-9a0e-523e2379f7ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 01:22:16.081099 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:16.080107    4864 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 01:22:16.081099 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:16.080166    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z" podUID="3aa394cb-eba6-4e15-9a0e-523e2379f7ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 01:22:17.066906 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:17.066875    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:17.067301 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:17.067266    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:17.067895 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:17.067876    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:22:17.749084 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:17.749041    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"
Oct 30 01:22:19.040507 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:22:19.040470    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:22:34.415193 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:34.415159    4864 patch_prober.go:29] interesting pod/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:22:34.415542 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:34.415210    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8" podUID="41fd0656-df2b-4cd7-9f6a-ac93a18f9579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:22:34.988408 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:34.988376    4864 generic.go:356] "Generic (PLEG): container finished" podID="41fd0656-df2b-4cd7-9f6a-ac93a18f9579" containerID="40a68cf616bf0a6e16e226ba8b6d9b729a35261b1f4d552168efcb17024f35bd" exitCode=0
Oct 30 01:22:34.988663 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:34.988453    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8" event={"ID":"41fd0656-df2b-4cd7-9f6a-ac93a18f9579","Type":"ContainerDied","Data":"40a68cf616bf0a6e16e226ba8b6d9b729a35261b1f4d552168efcb17024f35bd"}
Oct 30 01:22:34.988930 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:34.988912    4864 scope.go:122] "RemoveContainer" containerID="40a68cf616bf0a6e16e226ba8b6d9b729a35261b1f4d552168efcb17024f35bd"
Oct 30 01:22:34.990892 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:34.990858    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:22:34.990981 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:34.990948    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:22:34.991021 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:34.990992    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:22:36.015096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:36.004225    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8" event={"ID":"41fd0656-df2b-4cd7-9f6a-ac93a18f9579","Type":"ContainerStarted","Data":"173398fe52fbe34f9a33bbf6b10703c6c903b704cea5369ff43d6c2094c96645"}
Oct 30 01:22:36.875096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:36.873368    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:22:36.875096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:36.873491    4864 patch_prober.go:29] interesting pod/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 01:22:36.875096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:36.873549    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8" podUID="41fd0656-df2b-4cd7-9f6a-ac93a18f9579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 01:22:37.881302 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:37.881207    4864 patch_prober.go:29] interesting pod/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 01:22:37.881302 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:37.881287    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8" podUID="41fd0656-df2b-4cd7-9f6a-ac93a18f9579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 01:22:38.881214 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:38.876164    4864 patch_prober.go:29] interesting pod/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 01:22:38.904291 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:38.904248    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8" podUID="41fd0656-df2b-4cd7-9f6a-ac93a18f9579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 01:22:39.006100 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:39.005620    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl"]
Oct 30 01:22:39.018392 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:39.018355    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl"]
Oct 30 01:22:39.018754 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:39.018740    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl"
Oct 30 01:22:39.034263 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:39.033821    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-c7bba9-serving-cert\""
Oct 30 01:22:39.088096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:39.085872    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44072e00-e386-4e3e-9d8a-3087d4dc9c16-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl\" (UID: \"44072e00-e386-4e3e-9d8a-3087d4dc9c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl"
Oct 30 01:22:39.088096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:39.086053    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28x9q\" (UniqueName: \"kubernetes.io/projected/44072e00-e386-4e3e-9d8a-3087d4dc9c16-kube-api-access-28x9q\") pod \"maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl\" (UID: \"44072e00-e386-4e3e-9d8a-3087d4dc9c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl"
Oct 30 01:22:39.088096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:39.086138    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44072e00-e386-4e3e-9d8a-3087d4dc9c16-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl\" (UID: \"44072e00-e386-4e3e-9d8a-3087d4dc9c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl"
Oct 30 01:22:39.088096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:39.086258    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44072e00-e386-4e3e-9d8a-3087d4dc9c16-metrics-certs\") pod \"maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl\" (UID: \"44072e00-e386-4e3e-9d8a-3087d4dc9c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl"
Oct 30 01:22:39.187908 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:39.187824    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28x9q\" (UniqueName: \"kubernetes.io/projected/44072e00-e386-4e3e-9d8a-3087d4dc9c16-kube-api-access-28x9q\") pod \"maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl\" (UID: \"44072e00-e386-4e3e-9d8a-3087d4dc9c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl"
Oct 30 01:22:39.188184 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:39.188164    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44072e00-e386-4e3e-9d8a-3087d4dc9c16-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl\" (UID: \"44072e00-e386-4e3e-9d8a-3087d4dc9c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl"
Oct 30 01:22:39.188451 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:39.188428    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44072e00-e386-4e3e-9d8a-3087d4dc9c16-metrics-certs\") pod \"maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl\" (UID: \"44072e00-e386-4e3e-9d8a-3087d4dc9c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl"
Oct 30 01:22:39.188987 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:39.188958    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44072e00-e386-4e3e-9d8a-3087d4dc9c16-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl\" (UID: \"44072e00-e386-4e3e-9d8a-3087d4dc9c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl"
Oct 30 01:22:39.189095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:39.189036    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44072e00-e386-4e3e-9d8a-3087d4dc9c16-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl\" (UID: \"44072e00-e386-4e3e-9d8a-3087d4dc9c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl"
Oct 30 01:22:39.191385 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:39.191230    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44072e00-e386-4e3e-9d8a-3087d4dc9c16-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl\" (UID: \"44072e00-e386-4e3e-9d8a-3087d4dc9c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl"
Oct 30 01:22:39.191385 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:39.191368    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44072e00-e386-4e3e-9d8a-3087d4dc9c16-metrics-certs\") pod \"maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl\" (UID: \"44072e00-e386-4e3e-9d8a-3087d4dc9c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl"
Oct 30 01:22:39.211263 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:39.211238    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28x9q\" (UniqueName: \"kubernetes.io/projected/44072e00-e386-4e3e-9d8a-3087d4dc9c16-kube-api-access-28x9q\") pod \"maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl\" (UID: \"44072e00-e386-4e3e-9d8a-3087d4dc9c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl"
Oct 30 01:22:39.342317 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:39.342231    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl"
Oct 30 01:22:39.685220 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:39.685189    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl"]
Oct 30 01:22:39.686734 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:22:39.686706    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44072e00_e386_4e3e_9d8a_3087d4dc9c16.slice/crio-bf6f4fd9ab974f90c316eee6c45ecbd2475bf8aed047e1672d90098d738a9c46 WatchSource:0}: Error finding container bf6f4fd9ab974f90c316eee6c45ecbd2475bf8aed047e1672d90098d738a9c46: Status 404 returned error can't find the container with id bf6f4fd9ab974f90c316eee6c45ecbd2475bf8aed047e1672d90098d738a9c46
Oct 30 01:22:39.879096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:39.875828    4864 patch_prober.go:29] interesting pod/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 01:22:39.879096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:39.875899    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8" podUID="41fd0656-df2b-4cd7-9f6a-ac93a18f9579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 01:22:40.091668 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:40.091622    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl" event={"ID":"44072e00-e386-4e3e-9d8a-3087d4dc9c16","Type":"ContainerStarted","Data":"bf6f4fd9ab974f90c316eee6c45ecbd2475bf8aed047e1672d90098d738a9c46"}
Oct 30 01:22:40.879096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:40.878065    4864 patch_prober.go:29] interesting pod/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 01:22:40.879096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:40.878156    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8" podUID="41fd0656-df2b-4cd7-9f6a-ac93a18f9579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 01:22:41.099581 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:41.099546    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl" event={"ID":"44072e00-e386-4e3e-9d8a-3087d4dc9c16","Type":"ContainerStarted","Data":"047d33e798e5fd7ab02068475a674018cb184b317353f10780fc3403ab67901b"}
Oct 30 01:22:41.100045 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:41.099636    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl"
Oct 30 01:22:41.144177 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:41.143907    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl" podStartSLOduration=2.7205150639999998 podStartE2EDuration="3.143883577s" totalImagesPullingTime="423.368513ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:41.140012734 +0000 UTC m=+2473.478114657" watchObservedRunningTime="2026-10-30 01:22:41.143883577 +0000 UTC m=+2473.481985489"
Oct 30 01:22:41.877667 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:41.877636    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:22:41.877948 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:41.877925    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:22:41.878693 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:41.878674    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:22:47.111449 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:47.111421    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl"
Oct 30 01:22:49.043125 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:22:49.043061    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:22:53.161620 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.161585    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"]
Oct 30 01:22:53.168747 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.168693    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.189029 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.189002    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-ae1820-openshift-default-dockercfg-rc59d\""
Oct 30 01:22:53.202991 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.202956    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"]
Oct 30 01:22:53.299093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.294167    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778281fb-fe1b-4f27-bd43-62920826a0f9-istiod-ca-cert\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.299093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.294349    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-envoy\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.299093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.294450    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g69kx\" (UniqueName: \"kubernetes.io/projected/778281fb-fe1b-4f27-bd43-62920826a0f9-kube-api-access-g69kx\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.299093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.294550    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-credential-socket\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.299093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.294602    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-podinfo\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.299093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.294629    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-workload-certs\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.299093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.294664    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-workload-socket\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.299093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.294697    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-data\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.299093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.294737    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-token\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.396054 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.395968    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-credential-socket\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.396054 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.396038    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-podinfo\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.396392 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.396067    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-workload-certs\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.396392 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.396116    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-workload-socket\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.396392 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.396155    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-data\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.396392 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.396192    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-token\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.396392 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.396269    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778281fb-fe1b-4f27-bd43-62920826a0f9-istiod-ca-cert\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.396628 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.396401    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-envoy\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.396628 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.396452    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-credential-socket\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.396628 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.396497    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g69kx\" (UniqueName: \"kubernetes.io/projected/778281fb-fe1b-4f27-bd43-62920826a0f9-kube-api-access-g69kx\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.397174 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.397150    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-workload-socket\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.397693 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.397612    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-workload-certs\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.398091 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.398052    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-data\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.398409 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.398357    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778281fb-fe1b-4f27-bd43-62920826a0f9-istiod-ca-cert\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.400550 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.400525    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-envoy\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.401151 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.401128    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-podinfo\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.426140 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.426064    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g69kx\" (UniqueName: \"kubernetes.io/projected/778281fb-fe1b-4f27-bd43-62920826a0f9-kube-api-access-g69kx\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.433753 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.433723    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-token\") pod \"e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") " pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.485784 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.485754    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:53.838473 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.838439    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"]
Oct 30 01:22:53.840711 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:22:53.840682    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778281fb_fe1b_4f27_bd43_62920826a0f9.slice/crio-b1ee66ceb752b541bec097dcf52674fbbb7f14d0defe0bd0ffb1cd2980e6e171 WatchSource:0}: Error finding container b1ee66ceb752b541bec097dcf52674fbbb7f14d0defe0bd0ffb1cd2980e6e171: Status 404 returned error can't find the container with id b1ee66ceb752b541bec097dcf52674fbbb7f14d0defe0bd0ffb1cd2980e6e171
Oct 30 01:22:53.844112 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.844054    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:22:53.844206 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.844163    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:22:53.844251 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:53.844208    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:22:54.213401 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:54.213368    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh" event={"ID":"778281fb-fe1b-4f27-bd43-62920826a0f9","Type":"ContainerStarted","Data":"13165003d18702cd7e990b59a64ef4e23adc46333f81f36c2800deabe1c583df"}
Oct 30 01:22:54.213812 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:54.213427    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh" event={"ID":"778281fb-fe1b-4f27-bd43-62920826a0f9","Type":"ContainerStarted","Data":"b1ee66ceb752b541bec097dcf52674fbbb7f14d0defe0bd0ffb1cd2980e6e171"}
Oct 30 01:22:54.258817 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:54.258762    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh" podStartSLOduration=1.258746174 podStartE2EDuration="1.258746174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:54.25644617 +0000 UTC m=+2486.594548103" watchObservedRunningTime="2026-10-30 01:22:54.258746174 +0000 UTC m=+2486.596848083"
Oct 30 01:22:54.486822 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:54.486734    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:54.493334 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:54.493305    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:55.220222 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:55.220193    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:22:55.221167 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:22:55.221146    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:23:08.421136 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:08.421021    4864 generic.go:356] "Generic (PLEG): container finished" podID="3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" containerID="34f56cf60c8df35eae8740bda92891c5f88f45be1bb06a55c2b0ee36f44aa07f" exitCode=0
Oct 30 01:23:08.421136 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:08.421093    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4" event={"ID":"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c","Type":"ContainerDied","Data":"34f56cf60c8df35eae8740bda92891c5f88f45be1bb06a55c2b0ee36f44aa07f"}
Oct 30 01:23:08.421655 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:08.421511    4864 scope.go:122] "RemoveContainer" containerID="34f56cf60c8df35eae8740bda92891c5f88f45be1bb06a55c2b0ee36f44aa07f"
Oct 30 01:23:08.423229 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:08.423202    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:23:08.423320 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:08.423271    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:23:08.423320 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:08.423302    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:23:09.433836 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:09.433802    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4" event={"ID":"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c","Type":"ContainerStarted","Data":"2cf55743e6bde2042e678d38fbe8ebca98fb67325bfe10affa97104f39d419c8"}
Oct 30 01:23:10.398095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:10.394418    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:23:10.417280 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:10.417230    4864 patch_prober.go:29] interesting pod/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 01:23:10.417608 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:10.417547    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4" podUID="3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 01:23:11.392162 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:11.392118    4864 patch_prober.go:29] interesting pod/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 01:23:11.392695 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:11.392189    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4" podUID="3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 01:23:12.392082 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:12.392048    4864 patch_prober.go:29] interesting pod/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 01:23:12.392274 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:12.392141    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4" podUID="3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 01:23:13.392903 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:13.392867    4864 patch_prober.go:29] interesting pod/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 01:23:13.393364 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:13.392919    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4" podUID="3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 01:23:14.393143 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:14.393108    4864 patch_prober.go:29] interesting pod/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 01:23:14.393736 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:14.393695    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4" podUID="3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 01:23:15.397537 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:15.397505    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:23:15.398037 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:15.397900    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:23:15.398724 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:15.398705    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:23:19.039910 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:23:19.039870    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:23:25.494361 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.494311    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"]
Oct 30 01:23:25.499488 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.499462    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.517303 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.517282    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-94ea1e-openshift-default-dockercfg-nplbs\""
Oct 30 01:23:25.529351 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.529324    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"]
Oct 30 01:23:25.569680 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.569648    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-token\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.569835 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.569712    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd0afda-48df-44ec-bebd-52768b7bdbc0-istiod-ca-cert\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.569835 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.569776    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-workload-socket\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.569921 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.569844    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-credential-socket\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.569921 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.569875    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-data\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.570012 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.569987    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-workload-certs\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.570132 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.570118    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-podinfo\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.570191 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.570181    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97vtg\" (UniqueName: \"kubernetes.io/projected/afd0afda-48df-44ec-bebd-52768b7bdbc0-kube-api-access-97vtg\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.570232 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.570210    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-envoy\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.674364 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.674315    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-podinfo\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.674693 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.674667    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97vtg\" (UniqueName: \"kubernetes.io/projected/afd0afda-48df-44ec-bebd-52768b7bdbc0-kube-api-access-97vtg\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.674876 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.674848    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-envoy\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.675005 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.674944    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-token\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.675063 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.675047    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd0afda-48df-44ec-bebd-52768b7bdbc0-istiod-ca-cert\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.675173 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.675098    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-workload-socket\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.675235 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.675182    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-credential-socket\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.675294 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.675241    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-data\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.675355 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.675341    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-workload-certs\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.675821 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.675787    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-workload-certs\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.676812 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.676787    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-workload-socket\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.677500 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.677474    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-credential-socket\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.677796 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.677774    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd0afda-48df-44ec-bebd-52768b7bdbc0-istiod-ca-cert\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.677885 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.677809    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-data\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.716094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.697588    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-podinfo\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.716984 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.701791    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-token\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.717159 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.708965    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97vtg\" (UniqueName: \"kubernetes.io/projected/afd0afda-48df-44ec-bebd-52768b7bdbc0-kube-api-access-97vtg\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.726100 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.719544    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-envoy\") pod \"e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") " pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:25.814517 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:25.814434    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:26.134670 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:26.134644    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"]
Oct 30 01:23:26.135744 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:23:26.135717    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd0afda_48df_44ec_bebd_52768b7bdbc0.slice/crio-363684e05dcaaf50a4ece9278e9fde3e5324e871b829069d18c8295f190da86b WatchSource:0}: Error finding container 363684e05dcaaf50a4ece9278e9fde3e5324e871b829069d18c8295f190da86b: Status 404 returned error can't find the container with id 363684e05dcaaf50a4ece9278e9fde3e5324e871b829069d18c8295f190da86b
Oct 30 01:23:26.138609 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:26.138582    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:23:26.138686 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:26.138642    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:23:26.138724 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:26.138683    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:23:26.652543 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:26.652506    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7" event={"ID":"afd0afda-48df-44ec-bebd-52768b7bdbc0","Type":"ContainerStarted","Data":"9af37006555e0a21547d34f8b2eff55428ddccc6df1acbf519db4b089712b7dc"}
Oct 30 01:23:26.652943 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:26.652571    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7" event={"ID":"afd0afda-48df-44ec-bebd-52768b7bdbc0","Type":"ContainerStarted","Data":"363684e05dcaaf50a4ece9278e9fde3e5324e871b829069d18c8295f190da86b"}
Oct 30 01:23:26.707387 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:26.707336    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7" podStartSLOduration=1.7073234080000002 podStartE2EDuration="1.707323408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:26.704796648 +0000 UTC m=+2519.042898559" watchObservedRunningTime="2026-10-30 01:23:26.707323408 +0000 UTC m=+2519.045425317"
Oct 30 01:23:26.814885 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:26.814855    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:27.820848 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:27.820822    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:28.677668 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:28.677637    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:28.678657 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:28.678630    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:23:30.377745 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:30.377711    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"]
Oct 30 01:23:30.378270 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:30.377915    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6" podUID="19edd238-c752-42b1-8d42-05c558830fdd" containerName="istio-proxy" containerID="cri-o://1153d402d5fcad9ed8deeef6f21da1939cd64b94cd16ee90335ee2ada1430221" gracePeriod=30
Oct 30 01:23:35.654030 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.653947    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:23:35.760801 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.760761    4864 generic.go:356] "Generic (PLEG): container finished" podID="19edd238-c752-42b1-8d42-05c558830fdd" containerID="1153d402d5fcad9ed8deeef6f21da1939cd64b94cd16ee90335ee2ada1430221" exitCode=0
Oct 30 01:23:35.761004 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.760838    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"
Oct 30 01:23:35.761004 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.760967    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6" event={"ID":"19edd238-c752-42b1-8d42-05c558830fdd","Type":"ContainerDied","Data":"1153d402d5fcad9ed8deeef6f21da1939cd64b94cd16ee90335ee2ada1430221"}
Oct 30 01:23:35.761172 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.761009    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6" event={"ID":"19edd238-c752-42b1-8d42-05c558830fdd","Type":"ContainerDied","Data":"973f758d116b332ff89a75d5a948c925161057f288e7d69cc9345b62ef215570"}
Oct 30 01:23:35.761172 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.761037    4864 scope.go:122] "RemoveContainer" containerID="1153d402d5fcad9ed8deeef6f21da1939cd64b94cd16ee90335ee2ada1430221"
Oct 30 01:23:35.770294 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.770110    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-istio-envoy\") pod \"19edd238-c752-42b1-8d42-05c558830fdd\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") "
Oct 30 01:23:35.771686 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.771354    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-credential-socket\") pod \"19edd238-c752-42b1-8d42-05c558830fdd\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") "
Oct 30 01:23:35.771686 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.771487    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19edd238-c752-42b1-8d42-05c558830fdd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19edd238-c752-42b1-8d42-05c558830fdd-istiod-ca-cert\") pod \"19edd238-c752-42b1-8d42-05c558830fdd\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") "
Oct 30 01:23:35.771686 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.771591    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19edd238-c752-42b1-8d42-05c558830fdd-kube-api-access-8zphp\" (UniqueName: \"kubernetes.io/projected/19edd238-c752-42b1-8d42-05c558830fdd-kube-api-access-8zphp\") pod \"19edd238-c752-42b1-8d42-05c558830fdd\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") "
Oct 30 01:23:35.771686 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.771627    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19edd238-c752-42b1-8d42-05c558830fdd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19edd238-c752-42b1-8d42-05c558830fdd-istio-podinfo\") pod \"19edd238-c752-42b1-8d42-05c558830fdd\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") "
Oct 30 01:23:35.771686 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.771669    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-workload-certs\") pod \"19edd238-c752-42b1-8d42-05c558830fdd\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") "
Oct 30 01:23:35.772149 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.771729    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-istio-data\") pod \"19edd238-c752-42b1-8d42-05c558830fdd\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") "
Oct 30 01:23:35.772149 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.771811    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-workload-socket\") pod \"19edd238-c752-42b1-8d42-05c558830fdd\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") "
Oct 30 01:23:35.772149 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.771901    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19edd238-c752-42b1-8d42-05c558830fdd-istio-token\" (UniqueName: \"kubernetes.io/projected/19edd238-c752-42b1-8d42-05c558830fdd-istio-token\") pod \"19edd238-c752-42b1-8d42-05c558830fdd\" (UID: \"19edd238-c752-42b1-8d42-05c558830fdd\") "
Oct 30 01:23:35.777015 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.775191    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-istio-data" pod "19edd238-c752-42b1-8d42-05c558830fdd" (UID: "19edd238-c752-42b1-8d42-05c558830fdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:35.777015 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.775415    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-workload-certs" pod "19edd238-c752-42b1-8d42-05c558830fdd" (UID: "19edd238-c752-42b1-8d42-05c558830fdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:35.777015 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.775689    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19edd238-c752-42b1-8d42-05c558830fdd-istiod-ca-cert" pod "19edd238-c752-42b1-8d42-05c558830fdd" (UID: "19edd238-c752-42b1-8d42-05c558830fdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:35.777015 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.775931    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-credential-socket" pod "19edd238-c752-42b1-8d42-05c558830fdd" (UID: "19edd238-c752-42b1-8d42-05c558830fdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:35.777015 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.776497    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-workload-socket" pod "19edd238-c752-42b1-8d42-05c558830fdd" (UID: "19edd238-c752-42b1-8d42-05c558830fdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:35.782270 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.779532    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-istio-envoy" pod "19edd238-c752-42b1-8d42-05c558830fdd" (UID: "19edd238-c752-42b1-8d42-05c558830fdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:35.782270 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.780999    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19edd238-c752-42b1-8d42-05c558830fdd-istio-podinfo" pod "19edd238-c752-42b1-8d42-05c558830fdd" (UID: "19edd238-c752-42b1-8d42-05c558830fdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:23:35.782270 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.782234    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19edd238-c752-42b1-8d42-05c558830fdd-kube-api-access-8zphp" pod "19edd238-c752-42b1-8d42-05c558830fdd" (UID: "19edd238-c752-42b1-8d42-05c558830fdd"). InnerVolumeSpecName "kube-api-access-8zphp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:35.802479 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.800440    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19edd238-c752-42b1-8d42-05c558830fdd-istio-token" pod "19edd238-c752-42b1-8d42-05c558830fdd" (UID: "19edd238-c752-42b1-8d42-05c558830fdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:35.816051 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.816025    4864 scope.go:122] "RemoveContainer" containerID="1153d402d5fcad9ed8deeef6f21da1939cd64b94cd16ee90335ee2ada1430221"
Oct 30 01:23:35.816537 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:23:35.816513    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1153d402d5fcad9ed8deeef6f21da1939cd64b94cd16ee90335ee2ada1430221\": container with ID starting with 1153d402d5fcad9ed8deeef6f21da1939cd64b94cd16ee90335ee2ada1430221 not found: ID does not exist" containerID="1153d402d5fcad9ed8deeef6f21da1939cd64b94cd16ee90335ee2ada1430221"
Oct 30 01:23:35.816632 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.816549    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1153d402d5fcad9ed8deeef6f21da1939cd64b94cd16ee90335ee2ada1430221"} err="failed to get container status \"1153d402d5fcad9ed8deeef6f21da1939cd64b94cd16ee90335ee2ada1430221\": rpc error: code = NotFound desc = could not find container \"1153d402d5fcad9ed8deeef6f21da1939cd64b94cd16ee90335ee2ada1430221\": container with ID starting with 1153d402d5fcad9ed8deeef6f21da1939cd64b94cd16ee90335ee2ada1430221 not found: ID does not exist"
Oct 30 01:23:35.874831 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.873791    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:23:35.874831 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.873827    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:23:35.874831 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.873842    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19edd238-c752-42b1-8d42-05c558830fdd-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:23:35.874831 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.873855    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zphp\" (UniqueName: \"kubernetes.io/projected/19edd238-c752-42b1-8d42-05c558830fdd-kube-api-access-8zphp\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:23:35.874831 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.873868    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19edd238-c752-42b1-8d42-05c558830fdd-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:23:35.874831 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.873880    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:23:35.874831 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.873892    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:23:35.874831 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.873902    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19edd238-c752-42b1-8d42-05c558830fdd-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:23:35.874831 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:35.873916    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19edd238-c752-42b1-8d42-05c558830fdd-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:23:36.105744 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:36.105698    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"]
Oct 30 01:23:36.116226 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:36.116194    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-88lj6"]
Oct 30 01:23:37.801593 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:37.801559    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19edd238-c752-42b1-8d42-05c558830fdd" path="/var/lib/kubelet/pods/19edd238-c752-42b1-8d42-05c558830fdd/volumes"
Oct 30 01:23:37.826507 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:37.826471    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq"]
Oct 30 01:23:37.827143 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:37.827128    4864 cpu_manager.go:414] "containerMap: removing container" podUID="19edd238-c752-42b1-8d42-05c558830fdd" containerName="istio-proxy"
Oct 30 01:23:37.827209 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:37.827144    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19edd238-c752-42b1-8d42-05c558830fdd" containerName="istio-proxy"
Oct 30 01:23:37.827257 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:37.827248    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="19edd238-c752-42b1-8d42-05c558830fdd" containerName="istio-proxy"
Oct 30 01:23:37.832218 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:37.832197    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq"
Oct 30 01:23:37.848391 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:37.848357    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-1afb3e-serving-cert\""
Oct 30 01:23:37.862307 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:37.862280    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq"]
Oct 30 01:23:38.003552 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:38.003520    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z8xr\" (UniqueName: \"kubernetes.io/projected/22765564-41a8-4a68-9bd6-9aff4d7a0655-kube-api-access-8z8xr\") pod \"maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq\" (UID: \"22765564-41a8-4a68-9bd6-9aff4d7a0655\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq"
Oct 30 01:23:38.003737 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:38.003649    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22765564-41a8-4a68-9bd6-9aff4d7a0655-metrics-certs\") pod \"maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq\" (UID: \"22765564-41a8-4a68-9bd6-9aff4d7a0655\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq"
Oct 30 01:23:38.003785 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:38.003729    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22765564-41a8-4a68-9bd6-9aff4d7a0655-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq\" (UID: \"22765564-41a8-4a68-9bd6-9aff4d7a0655\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq"
Oct 30 01:23:38.003893 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:38.003874    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22765564-41a8-4a68-9bd6-9aff4d7a0655-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq\" (UID: \"22765564-41a8-4a68-9bd6-9aff4d7a0655\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq"
Oct 30 01:23:38.105662 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:38.105569    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z8xr\" (UniqueName: \"kubernetes.io/projected/22765564-41a8-4a68-9bd6-9aff4d7a0655-kube-api-access-8z8xr\") pod \"maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq\" (UID: \"22765564-41a8-4a68-9bd6-9aff4d7a0655\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq"
Oct 30 01:23:38.105662 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:38.105634    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22765564-41a8-4a68-9bd6-9aff4d7a0655-metrics-certs\") pod \"maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq\" (UID: \"22765564-41a8-4a68-9bd6-9aff4d7a0655\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq"
Oct 30 01:23:38.105921 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:38.105678    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22765564-41a8-4a68-9bd6-9aff4d7a0655-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq\" (UID: \"22765564-41a8-4a68-9bd6-9aff4d7a0655\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq"
Oct 30 01:23:38.105921 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:38.105758    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22765564-41a8-4a68-9bd6-9aff4d7a0655-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq\" (UID: \"22765564-41a8-4a68-9bd6-9aff4d7a0655\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq"
Oct 30 01:23:38.106688 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:38.106655    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22765564-41a8-4a68-9bd6-9aff4d7a0655-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq\" (UID: \"22765564-41a8-4a68-9bd6-9aff4d7a0655\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq"
Oct 30 01:23:38.108386 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:38.108360    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22765564-41a8-4a68-9bd6-9aff4d7a0655-metrics-certs\") pod \"maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq\" (UID: \"22765564-41a8-4a68-9bd6-9aff4d7a0655\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq"
Oct 30 01:23:38.108492 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:38.108477    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22765564-41a8-4a68-9bd6-9aff4d7a0655-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq\" (UID: \"22765564-41a8-4a68-9bd6-9aff4d7a0655\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq"
Oct 30 01:23:38.127268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:38.127240    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z8xr\" (UniqueName: \"kubernetes.io/projected/22765564-41a8-4a68-9bd6-9aff4d7a0655-kube-api-access-8z8xr\") pod \"maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq\" (UID: \"22765564-41a8-4a68-9bd6-9aff4d7a0655\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq"
Oct 30 01:23:38.146321 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:38.146292    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq"
Oct 30 01:23:38.461979 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:38.461948    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq"]
Oct 30 01:23:38.462994 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:23:38.462968    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22765564_41a8_4a68_9bd6_9aff4d7a0655.slice/crio-44db66e150e52ae53e455a715ece3ae5c6e054e16f6bcbff79b1a1a2ff1bc07f WatchSource:0}: Error finding container 44db66e150e52ae53e455a715ece3ae5c6e054e16f6bcbff79b1a1a2ff1bc07f: Status 404 returned error can't find the container with id 44db66e150e52ae53e455a715ece3ae5c6e054e16f6bcbff79b1a1a2ff1bc07f
Oct 30 01:23:38.782748 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:38.782700    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq" event={"ID":"22765564-41a8-4a68-9bd6-9aff4d7a0655","Type":"ContainerStarted","Data":"44db66e150e52ae53e455a715ece3ae5c6e054e16f6bcbff79b1a1a2ff1bc07f"}
Oct 30 01:23:39.799695 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:39.799659    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq" event={"ID":"22765564-41a8-4a68-9bd6-9aff4d7a0655","Type":"ContainerStarted","Data":"36f400150e34203146a7c4760cad0f3ce4d469ab424700bb3b2ebeab39504520"}
Oct 30 01:23:39.800123 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:39.799719    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq"
Oct 30 01:23:39.837510 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:39.837434    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq" podStartSLOduration=2.574586005 podStartE2EDuration="2.837416718s" totalImagesPullingTime="262.830713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:39.837225127 +0000 UTC m=+2532.175327038" watchObservedRunningTime="2026-10-30 01:23:39.837416718 +0000 UTC m=+2532.175518629"
Oct 30 01:23:45.810032 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:23:45.810005    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq"
Oct 30 01:23:49.040456 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:23:49.040423    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:24:09.028920 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:09.028829    4864 generic.go:356] "Generic (PLEG): container finished" podID="778281fb-fe1b-4f27-bd43-62920826a0f9" containerID="13165003d18702cd7e990b59a64ef4e23adc46333f81f36c2800deabe1c583df" exitCode=0
Oct 30 01:24:09.028920 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:09.028902    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh" event={"ID":"778281fb-fe1b-4f27-bd43-62920826a0f9","Type":"ContainerDied","Data":"13165003d18702cd7e990b59a64ef4e23adc46333f81f36c2800deabe1c583df"}
Oct 30 01:24:09.029532 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:09.029473    4864 scope.go:122] "RemoveContainer" containerID="13165003d18702cd7e990b59a64ef4e23adc46333f81f36c2800deabe1c583df"
Oct 30 01:24:09.031576 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:09.031547    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:24:09.031720 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:09.031702    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:24:09.031770 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:09.031756    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:24:10.049123 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:10.048998    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh" event={"ID":"778281fb-fe1b-4f27-bd43-62920826a0f9","Type":"ContainerStarted","Data":"64463487d47622c7848f48f30c8453b651583aed738acaca624ad5c79f94899d"}
Oct 30 01:24:10.220526 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:10.220489    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:24:10.486038 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:10.485998    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:24:10.487404 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:10.487378    4864 patch_prober.go:29] interesting pod/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 01:24:10.487521 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:10.487448    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh" podUID="778281fb-fe1b-4f27-bd43-62920826a0f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 01:24:11.487539 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:11.487502    4864 patch_prober.go:29] interesting pod/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 01:24:11.488054 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:11.487667    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh" podUID="778281fb-fe1b-4f27-bd43-62920826a0f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 01:24:12.486713 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:12.486674    4864 patch_prober.go:29] interesting pod/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 01:24:12.486910 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:12.486727    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh" podUID="778281fb-fe1b-4f27-bd43-62920826a0f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 01:24:13.489516 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:13.489478    4864 patch_prober.go:29] interesting pod/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 01:24:13.490547 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:13.490503    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh" podUID="778281fb-fe1b-4f27-bd43-62920826a0f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 01:24:14.487087 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:14.487038    4864 patch_prober.go:29] interesting pod/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 01:24:14.487270 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:14.487140    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh" podUID="778281fb-fe1b-4f27-bd43-62920826a0f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 01:24:15.566543 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:15.566507    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:24:15.568420 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:15.568382    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:24:19.041010 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:24:19.040969    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:24:44.726302 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:44.725409    4864 patch_prober.go:29] interesting pod/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:24:44.726302 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:44.725493    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7" podUID="afd0afda-48df-44ec-bebd-52768b7bdbc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:45.372343 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:45.372294    4864 generic.go:356] "Generic (PLEG): container finished" podID="afd0afda-48df-44ec-bebd-52768b7bdbc0" containerID="9af37006555e0a21547d34f8b2eff55428ddccc6df1acbf519db4b089712b7dc" exitCode=0
Oct 30 01:24:45.372568 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:45.372348    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7" event={"ID":"afd0afda-48df-44ec-bebd-52768b7bdbc0","Type":"ContainerDied","Data":"9af37006555e0a21547d34f8b2eff55428ddccc6df1acbf519db4b089712b7dc"}
Oct 30 01:24:45.372875 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:45.372853    4864 scope.go:122] "RemoveContainer" containerID="9af37006555e0a21547d34f8b2eff55428ddccc6df1acbf519db4b089712b7dc"
Oct 30 01:24:45.374848 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:45.374813    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:24:45.374968 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:45.374899    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:24:45.374968 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:45.374933    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:24:46.387864 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:46.387832    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7" event={"ID":"afd0afda-48df-44ec-bebd-52768b7bdbc0","Type":"ContainerStarted","Data":"b6ffe50c10590157c17ab77d7f60c99261c3bf12b9c54086e9288dbacbb2d004"}
Oct 30 01:24:46.815086 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:46.815053    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:24:46.815289 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:46.815239    4864 patch_prober.go:29] interesting pod/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 01:24:46.815368 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:46.815290    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7" podUID="afd0afda-48df-44ec-bebd-52768b7bdbc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 01:24:47.814870 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:47.814818    4864 patch_prober.go:29] interesting pod/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 01:24:47.815423 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:47.814891    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7" podUID="afd0afda-48df-44ec-bebd-52768b7bdbc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 01:24:48.814773 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:48.814734    4864 patch_prober.go:29] interesting pod/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 01:24:48.814978 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:48.814790    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7" podUID="afd0afda-48df-44ec-bebd-52768b7bdbc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 01:24:49.048439 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:24:49.048395    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:24:49.815563 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:49.815525    4864 patch_prober.go:29] interesting pod/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 01:24:49.815988 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:49.815590    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7" podUID="afd0afda-48df-44ec-bebd-52768b7bdbc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 01:24:50.815439 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:50.815398    4864 patch_prober.go:29] interesting pod/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 01:24:50.815624 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:50.815466    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7" podUID="afd0afda-48df-44ec-bebd-52768b7bdbc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 01:24:51.819711 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:51.819679    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:24:51.820121 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:51.820103    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:24:51.820827 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:24:51.820806    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:25:19.040771 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:25:19.040732    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:25:49.039928 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:25:49.039841    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:26:19.040760 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:26:19.040723    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:26:28.181763 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:28.181735    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:26:28.193977 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:28.193955    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:26:36.193283 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.193242    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"]
Oct 30 01:26:36.197712 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.197692    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.210698 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.210675    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-1cb829-openshift-default-dockercfg-gh8kg\""
Oct 30 01:26:36.226334 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.226304    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"]
Oct 30 01:26:36.267480 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.267444    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48828a7f-0845-4165-9046-40358f733ba1-istio-token\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.267662 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.267628    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48828a7f-0845-4165-9046-40358f733ba1-istio-podinfo\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.267715 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.267686    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slkmh\" (UniqueName: \"kubernetes.io/projected/48828a7f-0845-4165-9046-40358f733ba1-kube-api-access-slkmh\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.267800 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.267776    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-istio-envoy\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.267858 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.267842    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-workload-socket\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.267976 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.267955    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-workload-certs\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.268052 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.267998    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48828a7f-0845-4165-9046-40358f733ba1-istiod-ca-cert\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.268119 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.268052    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-istio-data\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.268228 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.268207    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-credential-socket\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.369254 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.369218    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-credential-socket\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.369518 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.369293    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48828a7f-0845-4165-9046-40358f733ba1-istio-token\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.369518 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.369349    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48828a7f-0845-4165-9046-40358f733ba1-istio-podinfo\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.369518 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.369373    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slkmh\" (UniqueName: \"kubernetes.io/projected/48828a7f-0845-4165-9046-40358f733ba1-kube-api-access-slkmh\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.369518 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.369394    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-istio-envoy\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.369518 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.369421    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-workload-socket\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.369518 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.369468    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-workload-certs\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.369518 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.369497    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48828a7f-0845-4165-9046-40358f733ba1-istiod-ca-cert\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.369859 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.369544    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-istio-data\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.369859 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.369698    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-credential-socket\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.369979 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.369864    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-workload-socket\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.369979 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.369956    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-workload-certs\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.370116 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.370057    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-istio-data\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.370239 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.370209    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48828a7f-0845-4165-9046-40358f733ba1-istiod-ca-cert\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.371672 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.371649    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-istio-envoy\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.371997 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.371972    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48828a7f-0845-4165-9046-40358f733ba1-istio-podinfo\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.390923 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.390901    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48828a7f-0845-4165-9046-40358f733ba1-istio-token\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.391281 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.391258    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slkmh\" (UniqueName: \"kubernetes.io/projected/48828a7f-0845-4165-9046-40358f733ba1-kube-api-access-slkmh\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.511929 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.511900    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:36.828152 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.828126    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"]
Oct 30 01:26:36.829362 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:26:36.829337    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48828a7f_0845_4165_9046_40358f733ba1.slice/crio-c34f400a76251a4441458d3c64154c1694c21024869bfa98358f68fe70d837ed WatchSource:0}: Error finding container c34f400a76251a4441458d3c64154c1694c21024869bfa98358f68fe70d837ed: Status 404 returned error can't find the container with id c34f400a76251a4441458d3c64154c1694c21024869bfa98358f68fe70d837ed
Oct 30 01:26:36.832055 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.832023    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:26:36.832152 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.832102    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:26:36.832152 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:36.832136    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:26:37.230971 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:37.230936    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk" event={"ID":"48828a7f-0845-4165-9046-40358f733ba1","Type":"ContainerStarted","Data":"fe3c310093366327ebe676da12b7fd5e05e57d473480a7aa373e29b184925043"}
Oct 30 01:26:37.231507 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:37.230990    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk" event={"ID":"48828a7f-0845-4165-9046-40358f733ba1","Type":"ContainerStarted","Data":"c34f400a76251a4441458d3c64154c1694c21024869bfa98358f68fe70d837ed"}
Oct 30 01:26:37.285524 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:37.284448    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk" podStartSLOduration=1.28442219 podStartE2EDuration="1.28442219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:37.282608987 +0000 UTC m=+2709.620710899" watchObservedRunningTime="2026-10-30 01:26:37.28442219 +0000 UTC m=+2709.622524100"
Oct 30 01:26:37.516564 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:37.516474    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:37.519810 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:37.519774    4864 patch_prober.go:29] interesting pod/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 01:26:37.520154 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:37.520109    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk" podUID="48828a7f-0845-4165-9046-40358f733ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 01:26:38.512376 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:38.512339    4864 patch_prober.go:29] interesting pod/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 01:26:38.512774 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:38.512392    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk" podUID="48828a7f-0845-4165-9046-40358f733ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 01:26:39.512970 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:39.512938    4864 patch_prober.go:29] interesting pod/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 01:26:39.513410 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:39.512992    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk" podUID="48828a7f-0845-4165-9046-40358f733ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 01:26:40.519885 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:40.519847    4864 patch_prober.go:29] interesting pod/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 01:26:40.520463 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:40.520432    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk" podUID="48828a7f-0845-4165-9046-40358f733ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 01:26:41.512942 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:41.512910    4864 patch_prober.go:29] interesting pod/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 01:26:41.513144 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:41.512961    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk" podUID="48828a7f-0845-4165-9046-40358f733ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 01:26:42.513205 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:42.513172    4864 patch_prober.go:29] interesting pod/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 01:26:42.513633 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:42.513228    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk" podUID="48828a7f-0845-4165-9046-40358f733ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 01:26:43.518139 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:43.518103    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:43.519118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:43.519092    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:43.519866 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:26:43.519843    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:26:49.040698 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:26:49.040658    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:27:19.091602 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:27:19.091484    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:27:49.040572 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:27:49.040533    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:28:19.039947 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:28:19.039907    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:28:49.038783 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:28:49.038708    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:29:19.074004 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:29:19.073960    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:29:49.042133 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:29:49.042091    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:30:00.213339 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.213236    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9"]
Oct 30 01:30:00.219917 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.219895    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9"
Oct 30 01:30:00.247397 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.247362    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg"]
Oct 30 01:30:00.258635 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.258607    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6"]
Oct 30 01:30:00.258799 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.258727    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg"
Oct 30 01:30:00.265853 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.265480    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:30:00.268011 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.267989    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9"]
Oct 30 01:30:00.268172 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.268150    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6"
Oct 30 01:30:00.273091 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.273036    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg"]
Oct 30 01:30:00.306929 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.306890    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn"]
Oct 30 01:30:00.316770 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.316736    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6"]
Oct 30 01:30:00.316941 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.316902    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn"
Oct 30 01:30:00.322407 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.322378    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx"]
Oct 30 01:30:00.340531 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.340486    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn"]
Oct 30 01:30:00.340671 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.340586    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx"
Oct 30 01:30:00.341724 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.341696    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx"]
Oct 30 01:30:00.369232 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.369208    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wfsh\" (UniqueName: \"kubernetes.io/projected/ce2530ca-6609-45c1-8438-17393a09e6c9-kube-api-access-6wfsh\") pod \"maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6\" (UID: \"ce2530ca-6609-45c1-8438-17393a09e6c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6"
Oct 30 01:30:00.369402 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.369383    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d653be-f408-4827-b79a-fba3572cc69d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg\" (UID: \"56d653be-f408-4827-b79a-fba3572cc69d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg"
Oct 30 01:30:00.369467 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.369453    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t92g\" (UniqueName: \"kubernetes.io/projected/40755b18-80e6-415f-8a27-4879ca6222b8-kube-api-access-9t92g\") pod \"maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9\" (UID: \"40755b18-80e6-415f-8a27-4879ca6222b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9"
Oct 30 01:30:00.369515 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.369504    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2530ca-6609-45c1-8438-17393a09e6c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6\" (UID: \"ce2530ca-6609-45c1-8438-17393a09e6c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6"
Oct 30 01:30:00.369608 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.369588    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl42s\" (UniqueName: \"kubernetes.io/projected/56d653be-f408-4827-b79a-fba3572cc69d-kube-api-access-xl42s\") pod \"maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg\" (UID: \"56d653be-f408-4827-b79a-fba3572cc69d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg"
Oct 30 01:30:00.369645 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.369624    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40755b18-80e6-415f-8a27-4879ca6222b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9\" (UID: \"40755b18-80e6-415f-8a27-4879ca6222b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9"
Oct 30 01:30:00.471184 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.471101    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wfsh\" (UniqueName: \"kubernetes.io/projected/ce2530ca-6609-45c1-8438-17393a09e6c9-kube-api-access-6wfsh\") pod \"maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6\" (UID: \"ce2530ca-6609-45c1-8438-17393a09e6c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6"
Oct 30 01:30:00.471334 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.471286    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d653be-f408-4827-b79a-fba3572cc69d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg\" (UID: \"56d653be-f408-4827-b79a-fba3572cc69d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg"
Oct 30 01:30:00.471334 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.471326    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94b230f0-95fe-42ce-8fdf-8fea818268e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx\" (UID: \"94b230f0-95fe-42ce-8fdf-8fea818268e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx"
Oct 30 01:30:00.471417 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.471390    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc4e617-7562-4f35-b084-aa303323866b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn\" (UID: \"abc4e617-7562-4f35-b084-aa303323866b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn"
Oct 30 01:30:00.471459 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.471431    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xl52\" (UniqueName: \"kubernetes.io/projected/abc4e617-7562-4f35-b084-aa303323866b-kube-api-access-4xl52\") pod \"maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn\" (UID: \"abc4e617-7562-4f35-b084-aa303323866b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn"
Oct 30 01:30:00.471494 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.471482    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t92g\" (UniqueName: \"kubernetes.io/projected/40755b18-80e6-415f-8a27-4879ca6222b8-kube-api-access-9t92g\") pod \"maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9\" (UID: \"40755b18-80e6-415f-8a27-4879ca6222b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9"
Oct 30 01:30:00.471588 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.471564    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f52vs\" (UniqueName: \"kubernetes.io/projected/94b230f0-95fe-42ce-8fdf-8fea818268e3-kube-api-access-f52vs\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx\" (UID: \"94b230f0-95fe-42ce-8fdf-8fea818268e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx"
Oct 30 01:30:00.471629 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.471619    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2530ca-6609-45c1-8438-17393a09e6c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6\" (UID: \"ce2530ca-6609-45c1-8438-17393a09e6c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6"
Oct 30 01:30:00.471713 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.471692    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl42s\" (UniqueName: \"kubernetes.io/projected/56d653be-f408-4827-b79a-fba3572cc69d-kube-api-access-xl42s\") pod \"maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg\" (UID: \"56d653be-f408-4827-b79a-fba3572cc69d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg"
Oct 30 01:30:00.471775 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.471732    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40755b18-80e6-415f-8a27-4879ca6222b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9\" (UID: \"40755b18-80e6-415f-8a27-4879ca6222b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9"
Oct 30 01:30:00.471936 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.471916    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d653be-f408-4827-b79a-fba3572cc69d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg\" (UID: \"56d653be-f408-4827-b79a-fba3572cc69d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg"
Oct 30 01:30:00.472326 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.472303    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2530ca-6609-45c1-8438-17393a09e6c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6\" (UID: \"ce2530ca-6609-45c1-8438-17393a09e6c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6"
Oct 30 01:30:00.472326 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.472321    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40755b18-80e6-415f-8a27-4879ca6222b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9\" (UID: \"40755b18-80e6-415f-8a27-4879ca6222b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9"
Oct 30 01:30:00.493018 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.492996    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wfsh\" (UniqueName: \"kubernetes.io/projected/ce2530ca-6609-45c1-8438-17393a09e6c9-kube-api-access-6wfsh\") pod \"maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6\" (UID: \"ce2530ca-6609-45c1-8438-17393a09e6c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6"
Oct 30 01:30:00.493299 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.493278    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t92g\" (UniqueName: \"kubernetes.io/projected/40755b18-80e6-415f-8a27-4879ca6222b8-kube-api-access-9t92g\") pod \"maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9\" (UID: \"40755b18-80e6-415f-8a27-4879ca6222b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9"
Oct 30 01:30:00.493358 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.493294    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl42s\" (UniqueName: \"kubernetes.io/projected/56d653be-f408-4827-b79a-fba3572cc69d-kube-api-access-xl42s\") pod \"maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg\" (UID: \"56d653be-f408-4827-b79a-fba3572cc69d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg"
Oct 30 01:30:00.556478 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.556448    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9"
Oct 30 01:30:00.586114 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.583232    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f52vs\" (UniqueName: \"kubernetes.io/projected/94b230f0-95fe-42ce-8fdf-8fea818268e3-kube-api-access-f52vs\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx\" (UID: \"94b230f0-95fe-42ce-8fdf-8fea818268e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx"
Oct 30 01:30:00.586114 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.583532    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94b230f0-95fe-42ce-8fdf-8fea818268e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx\" (UID: \"94b230f0-95fe-42ce-8fdf-8fea818268e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx"
Oct 30 01:30:00.586114 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.583598    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc4e617-7562-4f35-b084-aa303323866b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn\" (UID: \"abc4e617-7562-4f35-b084-aa303323866b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn"
Oct 30 01:30:00.586114 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.583637    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xl52\" (UniqueName: \"kubernetes.io/projected/abc4e617-7562-4f35-b084-aa303323866b-kube-api-access-4xl52\") pod \"maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn\" (UID: \"abc4e617-7562-4f35-b084-aa303323866b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn"
Oct 30 01:30:00.586114 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.584878    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94b230f0-95fe-42ce-8fdf-8fea818268e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx\" (UID: \"94b230f0-95fe-42ce-8fdf-8fea818268e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx"
Oct 30 01:30:00.586114 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.585629    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc4e617-7562-4f35-b084-aa303323866b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn\" (UID: \"abc4e617-7562-4f35-b084-aa303323866b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn"
Oct 30 01:30:00.599925 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.599358    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg"
Oct 30 01:30:00.613143 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.610934    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f52vs\" (UniqueName: \"kubernetes.io/projected/94b230f0-95fe-42ce-8fdf-8fea818268e3-kube-api-access-f52vs\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx\" (UID: \"94b230f0-95fe-42ce-8fdf-8fea818268e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx"
Oct 30 01:30:00.613350 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.613270    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6"
Oct 30 01:30:00.616173 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.616149    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xl52\" (UniqueName: \"kubernetes.io/projected/abc4e617-7562-4f35-b084-aa303323866b-kube-api-access-4xl52\") pod \"maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn\" (UID: \"abc4e617-7562-4f35-b084-aa303323866b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn"
Oct 30 01:30:00.643517 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.643292    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn"
Oct 30 01:30:00.658738 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.656921    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx"
Oct 30 01:30:00.906236 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:30:00.906193    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40755b18_80e6_415f_8a27_4879ca6222b8.slice/crio-2845cf76f38065871d552d6bdb885d027ac0ef4776df071b72e3f125ede461f5 WatchSource:0}: Error finding container 2845cf76f38065871d552d6bdb885d027ac0ef4776df071b72e3f125ede461f5: Status 404 returned error can't find the container with id 2845cf76f38065871d552d6bdb885d027ac0ef4776df071b72e3f125ede461f5
Oct 30 01:30:00.906826 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.906798    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9"]
Oct 30 01:30:00.909023 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.909002    4864 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:30:00.953477 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.953452    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6"]
Oct 30 01:30:00.954696 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:30:00.954664    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce2530ca_6609_45c1_8438_17393a09e6c9.slice/crio-e0b526f8851170249ff0cd16044c0b382a61968345328eaaaa21533bcf5786f3 WatchSource:0}: Error finding container e0b526f8851170249ff0cd16044c0b382a61968345328eaaaa21533bcf5786f3: Status 404 returned error can't find the container with id e0b526f8851170249ff0cd16044c0b382a61968345328eaaaa21533bcf5786f3
Oct 30 01:30:00.965290 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.965265    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg"]
Oct 30 01:30:00.965410 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:30:00.965332    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d653be_f408_4827_b79a_fba3572cc69d.slice/crio-c03886758b5fb3a5dd757939b115bea5a644424ac2db4351d8373b4d2cde9cd4 WatchSource:0}: Error finding container c03886758b5fb3a5dd757939b115bea5a644424ac2db4351d8373b4d2cde9cd4: Status 404 returned error can't find the container with id c03886758b5fb3a5dd757939b115bea5a644424ac2db4351d8373b4d2cde9cd4
Oct 30 01:30:00.999729 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:00.999644    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg" event={"ID":"56d653be-f408-4827-b79a-fba3572cc69d","Type":"ContainerStarted","Data":"c03886758b5fb3a5dd757939b115bea5a644424ac2db4351d8373b4d2cde9cd4"}
Oct 30 01:30:01.001586 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:01.001556    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6" event={"ID":"ce2530ca-6609-45c1-8438-17393a09e6c9","Type":"ContainerStarted","Data":"e0b526f8851170249ff0cd16044c0b382a61968345328eaaaa21533bcf5786f3"}
Oct 30 01:30:01.002620 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:01.002594    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn"]
Oct 30 01:30:01.002846 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:01.002822    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9" event={"ID":"40755b18-80e6-415f-8a27-4879ca6222b8","Type":"ContainerStarted","Data":"2845cf76f38065871d552d6bdb885d027ac0ef4776df071b72e3f125ede461f5"}
Oct 30 01:30:01.006396 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:30:01.006370    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc4e617_7562_4f35_b084_aa303323866b.slice/crio-f1cd5b88c643c5b7e2d829a85ccc0bb63190c5fb102ce27e2a8658697b0ae476 WatchSource:0}: Error finding container f1cd5b88c643c5b7e2d829a85ccc0bb63190c5fb102ce27e2a8658697b0ae476: Status 404 returned error can't find the container with id f1cd5b88c643c5b7e2d829a85ccc0bb63190c5fb102ce27e2a8658697b0ae476
Oct 30 01:30:01.012975 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:01.012953    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx"]
Oct 30 01:30:01.013734 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:30:01.013715    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b230f0_95fe_42ce_8fdf_8fea818268e3.slice/crio-68087f3294a639a918f81728d3f759f7965d0169ec4b0fc32caeb2b36edf5775 WatchSource:0}: Error finding container 68087f3294a639a918f81728d3f759f7965d0169ec4b0fc32caeb2b36edf5775: Status 404 returned error can't find the container with id 68087f3294a639a918f81728d3f759f7965d0169ec4b0fc32caeb2b36edf5775
Oct 30 01:30:02.012441 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:02.012399    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx" event={"ID":"94b230f0-95fe-42ce-8fdf-8fea818268e3","Type":"ContainerStarted","Data":"68087f3294a639a918f81728d3f759f7965d0169ec4b0fc32caeb2b36edf5775"}
Oct 30 01:30:02.014003 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:02.013955    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn" event={"ID":"abc4e617-7562-4f35-b084-aa303323866b","Type":"ContainerStarted","Data":"f1cd5b88c643c5b7e2d829a85ccc0bb63190c5fb102ce27e2a8658697b0ae476"}
Oct 30 01:30:05.476931 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:05.476898    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6702c2fe-tl2ld"]
Oct 30 01:30:05.481783 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:05.481765    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6702c2fe-tl2ld"
Oct 30 01:30:05.499372 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:05.499345    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6702c2fe-tl2ld"]
Oct 30 01:30:05.504342 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:05.504314    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:30:05.577157 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:05.577134    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea5291f2-fd1f-433f-8cd1-986568a5be4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6702c2fe-tl2ld\" (UID: \"ea5291f2-fd1f-433f-8cd1-986568a5be4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6702c2fe-tl2ld"
Oct 30 01:30:05.577270 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:05.577186    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8ft8\" (UniqueName: \"kubernetes.io/projected/ea5291f2-fd1f-433f-8cd1-986568a5be4e-kube-api-access-g8ft8\") pod \"maas-api-revoke-sub-6702c2fe-tl2ld\" (UID: \"ea5291f2-fd1f-433f-8cd1-986568a5be4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6702c2fe-tl2ld"
Oct 30 01:30:05.577320 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:05.577275    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea5291f2-fd1f-433f-8cd1-986568a5be4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6702c2fe-tl2ld\" (UID: \"ea5291f2-fd1f-433f-8cd1-986568a5be4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6702c2fe-tl2ld"
Oct 30 01:30:05.679129 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:05.679098    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea5291f2-fd1f-433f-8cd1-986568a5be4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6702c2fe-tl2ld\" (UID: \"ea5291f2-fd1f-433f-8cd1-986568a5be4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6702c2fe-tl2ld"
Oct 30 01:30:05.679270 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:05.679151    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8ft8\" (UniqueName: \"kubernetes.io/projected/ea5291f2-fd1f-433f-8cd1-986568a5be4e-kube-api-access-g8ft8\") pod \"maas-api-revoke-sub-6702c2fe-tl2ld\" (UID: \"ea5291f2-fd1f-433f-8cd1-986568a5be4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6702c2fe-tl2ld"
Oct 30 01:30:05.679270 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:05.679178    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea5291f2-fd1f-433f-8cd1-986568a5be4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6702c2fe-tl2ld\" (UID: \"ea5291f2-fd1f-433f-8cd1-986568a5be4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6702c2fe-tl2ld"
Oct 30 01:30:05.679853 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:05.679831    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea5291f2-fd1f-433f-8cd1-986568a5be4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6702c2fe-tl2ld\" (UID: \"ea5291f2-fd1f-433f-8cd1-986568a5be4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6702c2fe-tl2ld"
Oct 30 01:30:05.681563 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:05.681537    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea5291f2-fd1f-433f-8cd1-986568a5be4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6702c2fe-tl2ld\" (UID: \"ea5291f2-fd1f-433f-8cd1-986568a5be4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6702c2fe-tl2ld"
Oct 30 01:30:05.700336 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:05.700313    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8ft8\" (UniqueName: \"kubernetes.io/projected/ea5291f2-fd1f-433f-8cd1-986568a5be4e-kube-api-access-g8ft8\") pod \"maas-api-revoke-sub-6702c2fe-tl2ld\" (UID: \"ea5291f2-fd1f-433f-8cd1-986568a5be4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6702c2fe-tl2ld"
Oct 30 01:30:05.807558 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:05.807530    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6702c2fe-tl2ld"
Oct 30 01:30:06.074140 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:06.074108    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6" event={"ID":"ce2530ca-6609-45c1-8438-17393a09e6c9","Type":"ContainerStarted","Data":"064918308d2954127acd382c3c7874112cafb467d92dc683d6694961da9960a2"}
Oct 30 01:30:06.076838 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:06.076118    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9" event={"ID":"40755b18-80e6-415f-8a27-4879ca6222b8","Type":"ContainerStarted","Data":"fd80245a9f06d862b9dd209fa7d9dfbafd5d5d812bd17d0e12b55cdf193f2188"}
Oct 30 01:30:06.081114 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:06.081062    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg" event={"ID":"56d653be-f408-4827-b79a-fba3572cc69d","Type":"ContainerStarted","Data":"f8c3ed4eb2a5a05b0277d2891794f63a3213015261bfa2626bf524ecfe2ac9cd"}
Oct 30 01:30:06.132850 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:06.132815    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6702c2fe-tl2ld"]
Oct 30 01:30:06.133606 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:30:06.133574    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5291f2_fd1f_433f_8cd1_986568a5be4e.slice/crio-b9d183444e1b99ea02559156d745650366aa243ad5d0d7ab83d552a0d5f16b74 WatchSource:0}: Error finding container b9d183444e1b99ea02559156d745650366aa243ad5d0d7ab83d552a0d5f16b74: Status 404 returned error can't find the container with id b9d183444e1b99ea02559156d745650366aa243ad5d0d7ab83d552a0d5f16b74
Oct 30 01:30:06.135655 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:06.135608    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6" podStartSLOduration=1.712454124 podStartE2EDuration="6.135593578s" totalImagesPullingTime="4.423139454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:06.132807046 +0000 UTC m=+2918.470908958" watchObservedRunningTime="2026-10-30 01:30:06.135593578 +0000 UTC m=+2918.473695487"
Oct 30 01:30:06.174812 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:06.174759    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9" podStartSLOduration=1.7101416120000001 podStartE2EDuration="6.174743919s" totalImagesPullingTime="4.464602307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:06.170760827 +0000 UTC m=+2918.508862794" watchObservedRunningTime="2026-10-30 01:30:06.174743919 +0000 UTC m=+2918.512845829"
Oct 30 01:30:06.207605 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:06.207556    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg" podStartSLOduration=1.667397002 podStartE2EDuration="6.207540445s" totalImagesPullingTime="4.540143443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:06.205431501 +0000 UTC m=+2918.543533411" watchObservedRunningTime="2026-10-30 01:30:06.207540445 +0000 UTC m=+2918.545642383"
Oct 30 01:30:06.243340 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:06.243289    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx" podStartSLOduration=1.7769266350000001 podStartE2EDuration="6.243275449s" totalImagesPullingTime="4.466348814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:06.24186867 +0000 UTC m=+2918.579970594" watchObservedRunningTime="2026-10-30 01:30:06.243275449 +0000 UTC m=+2918.581377359"
Oct 30 01:30:07.089570 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:07.089537    4864 generic.go:356] "Generic (PLEG): container finished" podID="abc4e617-7562-4f35-b084-aa303323866b" containerID="9b95479c6c4b979e6b165a08757c968d30cca76feec23546fb7dcbf4ba496e62" exitCode=60
Oct 30 01:30:07.089986 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:07.089618    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn" event={"ID":"abc4e617-7562-4f35-b084-aa303323866b","Type":"ContainerDied","Data":"9b95479c6c4b979e6b165a08757c968d30cca76feec23546fb7dcbf4ba496e62"}
Oct 30 01:30:07.089986 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:07.089917    4864 scope.go:122] "RemoveContainer" containerID="9b95479c6c4b979e6b165a08757c968d30cca76feec23546fb7dcbf4ba496e62"
Oct 30 01:30:07.091475 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:07.091449    4864 generic.go:356] "Generic (PLEG): container finished" podID="94b230f0-95fe-42ce-8fdf-8fea818268e3" containerID="e92551c389b573e0b532e386e8e3a2c170af47964825d7dbaaad930e241d5d46" exitCode=60
Oct 30 01:30:07.091592 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:07.091477    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx" event={"ID":"94b230f0-95fe-42ce-8fdf-8fea818268e3","Type":"ContainerDied","Data":"e92551c389b573e0b532e386e8e3a2c170af47964825d7dbaaad930e241d5d46"}
Oct 30 01:30:07.091858 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:07.091842    4864 scope.go:122] "RemoveContainer" containerID="e92551c389b573e0b532e386e8e3a2c170af47964825d7dbaaad930e241d5d46"
Oct 30 01:30:07.093461 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:07.093436    4864 generic.go:356] "Generic (PLEG): container finished" podID="56d653be-f408-4827-b79a-fba3572cc69d" containerID="f8c3ed4eb2a5a05b0277d2891794f63a3213015261bfa2626bf524ecfe2ac9cd" exitCode=60
Oct 30 01:30:07.093567 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:07.093509    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg" event={"ID":"56d653be-f408-4827-b79a-fba3572cc69d","Type":"ContainerDied","Data":"f8c3ed4eb2a5a05b0277d2891794f63a3213015261bfa2626bf524ecfe2ac9cd"}
Oct 30 01:30:07.094015 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:07.093697    4864 scope.go:122] "RemoveContainer" containerID="f8c3ed4eb2a5a05b0277d2891794f63a3213015261bfa2626bf524ecfe2ac9cd"
Oct 30 01:30:07.095409 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:07.095385    4864 generic.go:356] "Generic (PLEG): container finished" podID="ce2530ca-6609-45c1-8438-17393a09e6c9" containerID="064918308d2954127acd382c3c7874112cafb467d92dc683d6694961da9960a2" exitCode=60
Oct 30 01:30:07.095752 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:07.095733    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6" event={"ID":"ce2530ca-6609-45c1-8438-17393a09e6c9","Type":"ContainerDied","Data":"064918308d2954127acd382c3c7874112cafb467d92dc683d6694961da9960a2"}
Oct 30 01:30:07.095845 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:07.095764    4864 scope.go:122] "RemoveContainer" containerID="064918308d2954127acd382c3c7874112cafb467d92dc683d6694961da9960a2"
Oct 30 01:30:07.097625 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:07.097603    4864 generic.go:356] "Generic (PLEG): container finished" podID="40755b18-80e6-415f-8a27-4879ca6222b8" containerID="fd80245a9f06d862b9dd209fa7d9dfbafd5d5d812bd17d0e12b55cdf193f2188" exitCode=60
Oct 30 01:30:07.097745 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:07.097681    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9" event={"ID":"40755b18-80e6-415f-8a27-4879ca6222b8","Type":"ContainerDied","Data":"fd80245a9f06d862b9dd209fa7d9dfbafd5d5d812bd17d0e12b55cdf193f2188"}
Oct 30 01:30:07.097875 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:07.097860    4864 scope.go:122] "RemoveContainer" containerID="fd80245a9f06d862b9dd209fa7d9dfbafd5d5d812bd17d0e12b55cdf193f2188"
Oct 30 01:30:07.099467 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:07.099450    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6702c2fe-tl2ld" event={"ID":"ea5291f2-fd1f-433f-8cd1-986568a5be4e","Type":"ContainerStarted","Data":"6e9afc1954aa7c4dc0a5efa501b2e49a2135ce114e1bf0babbe85c04e6bb2353"}
Oct 30 01:30:07.099552 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:07.099481    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6702c2fe-tl2ld" event={"ID":"ea5291f2-fd1f-433f-8cd1-986568a5be4e","Type":"ContainerStarted","Data":"b9d183444e1b99ea02559156d745650366aa243ad5d0d7ab83d552a0d5f16b74"}
Oct 30 01:30:07.251401 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:07.251351    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6702c2fe-tl2ld" podStartSLOduration=2.251337809 podStartE2EDuration="2.251337809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:07.247289391 +0000 UTC m=+2919.585391301" watchObservedRunningTime="2026-10-30 01:30:07.251337809 +0000 UTC m=+2919.589439719"
Oct 30 01:30:08.107904 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:08.107871    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6" event={"ID":"ce2530ca-6609-45c1-8438-17393a09e6c9","Type":"ContainerStarted","Data":"8ac5c54282b70219ca162a1546bea1ed5cd52b304e43e2e84a62a28f66397e58"}
Oct 30 01:30:08.109857 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:08.109828    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9" event={"ID":"40755b18-80e6-415f-8a27-4879ca6222b8","Type":"ContainerStarted","Data":"4dc150c259098b944953b8e20ad2b3b00485da810c0610cc81d7d46f1c0be984"}
Oct 30 01:30:08.111315 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:08.111291    4864 generic.go:356] "Generic (PLEG): container finished" podID="ea5291f2-fd1f-433f-8cd1-986568a5be4e" containerID="6e9afc1954aa7c4dc0a5efa501b2e49a2135ce114e1bf0babbe85c04e6bb2353" exitCode=0
Oct 30 01:30:08.111413 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:08.111361    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6702c2fe-tl2ld" event={"ID":"ea5291f2-fd1f-433f-8cd1-986568a5be4e","Type":"ContainerDied","Data":"6e9afc1954aa7c4dc0a5efa501b2e49a2135ce114e1bf0babbe85c04e6bb2353"}
Oct 30 01:30:08.113266 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:08.113244    4864 generic.go:356] "Generic (PLEG): container finished" podID="abc4e617-7562-4f35-b084-aa303323866b" containerID="8a42c6b0e3c375eb724b803e25fb011a6093810c3856b7a287830eb2139e20a7" exitCode=60
Oct 30 01:30:08.113343 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:08.113320    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn" event={"ID":"abc4e617-7562-4f35-b084-aa303323866b","Type":"ContainerDied","Data":"8a42c6b0e3c375eb724b803e25fb011a6093810c3856b7a287830eb2139e20a7"}
Oct 30 01:30:08.113395 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:08.113370    4864 scope.go:122] "RemoveContainer" containerID="9b95479c6c4b979e6b165a08757c968d30cca76feec23546fb7dcbf4ba496e62"
Oct 30 01:30:08.113548 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:08.113525    4864 scope.go:122] "RemoveContainer" containerID="8a42c6b0e3c375eb724b803e25fb011a6093810c3856b7a287830eb2139e20a7"
Oct 30 01:30:08.113830 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:08.113810    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn_odh-ai-gateway-infra(abc4e617-7562-4f35-b084-aa303323866b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn" podUID="abc4e617-7562-4f35-b084-aa303323866b"
Oct 30 01:30:08.115356 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:08.115332    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx" event={"ID":"94b230f0-95fe-42ce-8fdf-8fea818268e3","Type":"ContainerStarted","Data":"90078b4c869f67250d72c32fbf181eec5f4395137e3546b974be3cc4c340a33d"}
Oct 30 01:30:08.117335 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:08.117312    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg" event={"ID":"56d653be-f408-4827-b79a-fba3572cc69d","Type":"ContainerStarted","Data":"327ac44aab278ddb38f0d3bb2b51f640483653afbbb0cff6e4e45da14788dd1b"}
Oct 30 01:30:09.126030 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.125996    4864 generic.go:356] "Generic (PLEG): container finished" podID="ce2530ca-6609-45c1-8438-17393a09e6c9" containerID="8ac5c54282b70219ca162a1546bea1ed5cd52b304e43e2e84a62a28f66397e58" exitCode=60
Oct 30 01:30:09.126532 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.126107    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6" event={"ID":"ce2530ca-6609-45c1-8438-17393a09e6c9","Type":"ContainerDied","Data":"8ac5c54282b70219ca162a1546bea1ed5cd52b304e43e2e84a62a28f66397e58"}
Oct 30 01:30:09.126532 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.126183    4864 scope.go:122] "RemoveContainer" containerID="064918308d2954127acd382c3c7874112cafb467d92dc683d6694961da9960a2"
Oct 30 01:30:09.126532 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.126411    4864 scope.go:122] "RemoveContainer" containerID="8ac5c54282b70219ca162a1546bea1ed5cd52b304e43e2e84a62a28f66397e58"
Oct 30 01:30:09.126804 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:09.126778    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6_odh-ai-gateway-infra(ce2530ca-6609-45c1-8438-17393a09e6c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6" podUID="ce2530ca-6609-45c1-8438-17393a09e6c9"
Oct 30 01:30:09.128370 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.128347    4864 generic.go:356] "Generic (PLEG): container finished" podID="40755b18-80e6-415f-8a27-4879ca6222b8" containerID="4dc150c259098b944953b8e20ad2b3b00485da810c0610cc81d7d46f1c0be984" exitCode=60
Oct 30 01:30:09.128447 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.128425    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9" event={"ID":"40755b18-80e6-415f-8a27-4879ca6222b8","Type":"ContainerDied","Data":"4dc150c259098b944953b8e20ad2b3b00485da810c0610cc81d7d46f1c0be984"}
Oct 30 01:30:09.128666 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.128648    4864 scope.go:122] "RemoveContainer" containerID="4dc150c259098b944953b8e20ad2b3b00485da810c0610cc81d7d46f1c0be984"
Oct 30 01:30:09.128968 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:09.128948    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9_odh-ai-gateway-infra(40755b18-80e6-415f-8a27-4879ca6222b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9" podUID="40755b18-80e6-415f-8a27-4879ca6222b8"
Oct 30 01:30:09.130412 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.130388    4864 scope.go:122] "RemoveContainer" containerID="8a42c6b0e3c375eb724b803e25fb011a6093810c3856b7a287830eb2139e20a7"
Oct 30 01:30:09.130747 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:09.130727    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn_odh-ai-gateway-infra(abc4e617-7562-4f35-b084-aa303323866b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn" podUID="abc4e617-7562-4f35-b084-aa303323866b"
Oct 30 01:30:09.132131 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.132110    4864 generic.go:356] "Generic (PLEG): container finished" podID="94b230f0-95fe-42ce-8fdf-8fea818268e3" containerID="90078b4c869f67250d72c32fbf181eec5f4395137e3546b974be3cc4c340a33d" exitCode=60
Oct 30 01:30:09.132250 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.132186    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx" event={"ID":"94b230f0-95fe-42ce-8fdf-8fea818268e3","Type":"ContainerDied","Data":"90078b4c869f67250d72c32fbf181eec5f4395137e3546b974be3cc4c340a33d"}
Oct 30 01:30:09.132508 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.132491    4864 scope.go:122] "RemoveContainer" containerID="90078b4c869f67250d72c32fbf181eec5f4395137e3546b974be3cc4c340a33d"
Oct 30 01:30:09.132743 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:09.132719    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx_odh-ai-gateway-infra(94b230f0-95fe-42ce-8fdf-8fea818268e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx" podUID="94b230f0-95fe-42ce-8fdf-8fea818268e3"
Oct 30 01:30:09.133968 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.133945    4864 generic.go:356] "Generic (PLEG): container finished" podID="56d653be-f408-4827-b79a-fba3572cc69d" containerID="327ac44aab278ddb38f0d3bb2b51f640483653afbbb0cff6e4e45da14788dd1b" exitCode=60
Oct 30 01:30:09.134126 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.134011    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg" event={"ID":"56d653be-f408-4827-b79a-fba3572cc69d","Type":"ContainerDied","Data":"327ac44aab278ddb38f0d3bb2b51f640483653afbbb0cff6e4e45da14788dd1b"}
Oct 30 01:30:09.134240 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.134227    4864 scope.go:122] "RemoveContainer" containerID="327ac44aab278ddb38f0d3bb2b51f640483653afbbb0cff6e4e45da14788dd1b"
Oct 30 01:30:09.134575 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:09.134434    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg_odh-ai-gateway-infra(56d653be-f408-4827-b79a-fba3572cc69d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg" podUID="56d653be-f408-4827-b79a-fba3572cc69d"
Oct 30 01:30:09.141295 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.141269    4864 scope.go:122] "RemoveContainer" containerID="fd80245a9f06d862b9dd209fa7d9dfbafd5d5d812bd17d0e12b55cdf193f2188"
Oct 30 01:30:09.159851 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.159821    4864 scope.go:122] "RemoveContainer" containerID="e92551c389b573e0b532e386e8e3a2c170af47964825d7dbaaad930e241d5d46"
Oct 30 01:30:09.182305 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.182280    4864 scope.go:122] "RemoveContainer" containerID="f8c3ed4eb2a5a05b0277d2891794f63a3213015261bfa2626bf524ecfe2ac9cd"
Oct 30 01:30:09.310482 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.310456    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6702c2fe-tl2ld"
Oct 30 01:30:09.449821 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.449730    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea5291f2-fd1f-433f-8cd1-986568a5be4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea5291f2-fd1f-433f-8cd1-986568a5be4e-maas-api-xks-ca\") pod \"ea5291f2-fd1f-433f-8cd1-986568a5be4e\" (UID: \"ea5291f2-fd1f-433f-8cd1-986568a5be4e\") "
Oct 30 01:30:09.449821 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.449801    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea5291f2-fd1f-433f-8cd1-986568a5be4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea5291f2-fd1f-433f-8cd1-986568a5be4e-maas-api-openshift-ca\") pod \"ea5291f2-fd1f-433f-8cd1-986568a5be4e\" (UID: \"ea5291f2-fd1f-433f-8cd1-986568a5be4e\") "
Oct 30 01:30:09.450007 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.449981    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5291f2-fd1f-433f-8cd1-986568a5be4e-kube-api-access-g8ft8\" (UniqueName: \"kubernetes.io/projected/ea5291f2-fd1f-433f-8cd1-986568a5be4e-kube-api-access-g8ft8\") pod \"ea5291f2-fd1f-433f-8cd1-986568a5be4e\" (UID: \"ea5291f2-fd1f-433f-8cd1-986568a5be4e\") "
Oct 30 01:30:09.450191 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.450166    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea5291f2-fd1f-433f-8cd1-986568a5be4e-maas-api-openshift-ca" pod "ea5291f2-fd1f-433f-8cd1-986568a5be4e" (UID: "ea5291f2-fd1f-433f-8cd1-986568a5be4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:09.450471 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.450455    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea5291f2-fd1f-433f-8cd1-986568a5be4e-maas-api-openshift-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:30:09.451852 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.451829    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea5291f2-fd1f-433f-8cd1-986568a5be4e-maas-api-xks-ca" pod "ea5291f2-fd1f-433f-8cd1-986568a5be4e" (UID: "ea5291f2-fd1f-433f-8cd1-986568a5be4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:30:09.452053 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.452033    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5291f2-fd1f-433f-8cd1-986568a5be4e-kube-api-access-g8ft8" pod "ea5291f2-fd1f-433f-8cd1-986568a5be4e" (UID: "ea5291f2-fd1f-433f-8cd1-986568a5be4e"). InnerVolumeSpecName "kube-api-access-g8ft8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:09.551859 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.551822    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea5291f2-fd1f-433f-8cd1-986568a5be4e-maas-api-xks-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:30:09.551859 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:09.551860    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8ft8\" (UniqueName: \"kubernetes.io/projected/ea5291f2-fd1f-433f-8cd1-986568a5be4e-kube-api-access-g8ft8\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:30:10.142330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:10.142246    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6702c2fe-tl2ld"
Oct 30 01:30:10.142330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:10.142266    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6702c2fe-tl2ld" event={"ID":"ea5291f2-fd1f-433f-8cd1-986568a5be4e","Type":"ContainerDied","Data":"b9d183444e1b99ea02559156d745650366aa243ad5d0d7ab83d552a0d5f16b74"}
Oct 30 01:30:10.142784 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:10.142356    4864 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9d183444e1b99ea02559156d745650366aa243ad5d0d7ab83d552a0d5f16b74"
Oct 30 01:30:10.146112 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:10.146062    4864 scope.go:122] "RemoveContainer" containerID="327ac44aab278ddb38f0d3bb2b51f640483653afbbb0cff6e4e45da14788dd1b"
Oct 30 01:30:10.146420 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:10.146397    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg_odh-ai-gateway-infra(56d653be-f408-4827-b79a-fba3572cc69d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg" podUID="56d653be-f408-4827-b79a-fba3572cc69d"
Oct 30 01:30:10.147646 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:10.147623    4864 scope.go:122] "RemoveContainer" containerID="8ac5c54282b70219ca162a1546bea1ed5cd52b304e43e2e84a62a28f66397e58"
Oct 30 01:30:10.147882 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:10.147863    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6_odh-ai-gateway-infra(ce2530ca-6609-45c1-8438-17393a09e6c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6" podUID="ce2530ca-6609-45c1-8438-17393a09e6c9"
Oct 30 01:30:10.149067 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:10.149052    4864 scope.go:122] "RemoveContainer" containerID="4dc150c259098b944953b8e20ad2b3b00485da810c0610cc81d7d46f1c0be984"
Oct 30 01:30:10.149291 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:10.149274    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9_odh-ai-gateway-infra(40755b18-80e6-415f-8a27-4879ca6222b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9" podUID="40755b18-80e6-415f-8a27-4879ca6222b8"
Oct 30 01:30:10.916384 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:10.916352    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l"]
Oct 30 01:30:10.917057 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:10.917044    4864 cpu_manager.go:414] "containerMap: removing container" podUID="ea5291f2-fd1f-433f-8cd1-986568a5be4e" containerName="revoke-keys"
Oct 30 01:30:10.917116 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:10.917059    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea5291f2-fd1f-433f-8cd1-986568a5be4e" containerName="revoke-keys"
Oct 30 01:30:10.917182 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:10.917173    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea5291f2-fd1f-433f-8cd1-986568a5be4e" containerName="revoke-keys"
Oct 30 01:30:10.921210 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:10.921190    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l"
Oct 30 01:30:10.937298 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:10.937271    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 01:30:10.937429 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:10.937335    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 01:30:10.937429 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:10.937337    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-0e3c0a7229d2\""
Oct 30 01:30:10.939794 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:10.939779    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-06d7d794cf59\""
Oct 30 01:30:10.947962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:10.947936    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-drckg\""
Oct 30 01:30:10.952012 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:10.951977    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l"]
Oct 30 01:30:10.973952 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:10.973920    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xszpz\" (UniqueName: \"kubernetes.io/projected/8dffeea0-7e54-4c87-ba16-d3450b6403c2-kube-api-access-xszpz\") pod \"payload-processing-external-model-79c9fb4bbb-6vx9l\" (UID: \"8dffeea0-7e54-4c87-ba16-d3450b6403c2\") " pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l"
Oct 30 01:30:10.974161 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:10.974112    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dffeea0-7e54-4c87-ba16-d3450b6403c2-plugins-config-volume\") pod \"payload-processing-external-model-79c9fb4bbb-6vx9l\" (UID: \"8dffeea0-7e54-4c87-ba16-d3450b6403c2\") " pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l"
Oct 30 01:30:10.974266 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:10.974248    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8dffeea0-7e54-4c87-ba16-d3450b6403c2-routing-overlay-volume\") pod \"payload-processing-external-model-79c9fb4bbb-6vx9l\" (UID: \"8dffeea0-7e54-4c87-ba16-d3450b6403c2\") " pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l"
Oct 30 01:30:10.974340 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:10.974326    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8dffeea0-7e54-4c87-ba16-d3450b6403c2-provider-credentials\") pod \"payload-processing-external-model-79c9fb4bbb-6vx9l\" (UID: \"8dffeea0-7e54-4c87-ba16-d3450b6403c2\") " pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l"
Oct 30 01:30:10.992915 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:10.992887    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 01:30:10.993055 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:10.992912    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 01:30:11.076083 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:11.076048    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8dffeea0-7e54-4c87-ba16-d3450b6403c2-routing-overlay-volume\") pod \"payload-processing-external-model-79c9fb4bbb-6vx9l\" (UID: \"8dffeea0-7e54-4c87-ba16-d3450b6403c2\") " pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l"
Oct 30 01:30:11.076261 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:11.076151    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8dffeea0-7e54-4c87-ba16-d3450b6403c2-provider-credentials\") pod \"payload-processing-external-model-79c9fb4bbb-6vx9l\" (UID: \"8dffeea0-7e54-4c87-ba16-d3450b6403c2\") " pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l"
Oct 30 01:30:11.076261 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:11.076237    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xszpz\" (UniqueName: \"kubernetes.io/projected/8dffeea0-7e54-4c87-ba16-d3450b6403c2-kube-api-access-xszpz\") pod \"payload-processing-external-model-79c9fb4bbb-6vx9l\" (UID: \"8dffeea0-7e54-4c87-ba16-d3450b6403c2\") " pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l"
Oct 30 01:30:11.076399 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:11.076337    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dffeea0-7e54-4c87-ba16-d3450b6403c2-plugins-config-volume\") pod \"payload-processing-external-model-79c9fb4bbb-6vx9l\" (UID: \"8dffeea0-7e54-4c87-ba16-d3450b6403c2\") " pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l"
Oct 30 01:30:11.076838 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:11.076815    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8dffeea0-7e54-4c87-ba16-d3450b6403c2-routing-overlay-volume\") pod \"payload-processing-external-model-79c9fb4bbb-6vx9l\" (UID: \"8dffeea0-7e54-4c87-ba16-d3450b6403c2\") " pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l"
Oct 30 01:30:11.077158 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:11.077138    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dffeea0-7e54-4c87-ba16-d3450b6403c2-plugins-config-volume\") pod \"payload-processing-external-model-79c9fb4bbb-6vx9l\" (UID: \"8dffeea0-7e54-4c87-ba16-d3450b6403c2\") " pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l"
Oct 30 01:30:11.078763 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:11.078746    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8dffeea0-7e54-4c87-ba16-d3450b6403c2-provider-credentials\") pod \"payload-processing-external-model-79c9fb4bbb-6vx9l\" (UID: \"8dffeea0-7e54-4c87-ba16-d3450b6403c2\") " pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l"
Oct 30 01:30:11.099962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:11.099929    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xszpz\" (UniqueName: \"kubernetes.io/projected/8dffeea0-7e54-4c87-ba16-d3450b6403c2-kube-api-access-xszpz\") pod \"payload-processing-external-model-79c9fb4bbb-6vx9l\" (UID: \"8dffeea0-7e54-4c87-ba16-d3450b6403c2\") " pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l"
Oct 30 01:30:11.246097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:11.239404    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l"
Oct 30 01:30:11.636230 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:11.635698    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l"]
Oct 30 01:30:11.656699 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:30:11.656659    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dffeea0_7e54_4c87_ba16_d3450b6403c2.slice/crio-a740122fc1de5ee64000788bc48826b90df55aca7f7d416366f0871370aeb9da WatchSource:0}: Error finding container a740122fc1de5ee64000788bc48826b90df55aca7f7d416366f0871370aeb9da: Status 404 returned error can't find the container with id a740122fc1de5ee64000788bc48826b90df55aca7f7d416366f0871370aeb9da
Oct 30 01:30:12.165817 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:12.165774    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l" event={"ID":"8dffeea0-7e54-4c87-ba16-d3450b6403c2","Type":"ContainerStarted","Data":"f709e7c51a4c31ed8b1b8e4901ed6e2b2a4cb48e4c86088ecab1134462ec2517"}
Oct 30 01:30:12.166014 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:12.165845    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l" event={"ID":"8dffeea0-7e54-4c87-ba16-d3450b6403c2","Type":"ContainerStarted","Data":"a740122fc1de5ee64000788bc48826b90df55aca7f7d416366f0871370aeb9da"}
Oct 30 01:30:12.166014 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:12.165992    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l"
Oct 30 01:30:12.233483 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:12.233408    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l" podStartSLOduration=2.2333866430000002 podStartE2EDuration="2.233386643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:12.208382143 +0000 UTC m=+2924.546484053" watchObservedRunningTime="2026-10-30 01:30:12.233386643 +0000 UTC m=+2924.571488554"
Oct 30 01:30:12.323840 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:12.321735    4864 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0e3c0a7229d2: secret "agc-cred-0e3c0a7229d2" not found
Oct 30 01:30:12.323840 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:12.321767    4864 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l: secret "agc-cred-0e3c0a7229d2" not found
Oct 30 01:30:12.323840 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:12.321858    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dffeea0-7e54-4c87-ba16-d3450b6403c2-provider-credentials podName:8dffeea0-7e54-4c87-ba16-d3450b6403c2 nodeName:}" failed. No retries permitted until 2026-10-30 01:30:12.821826636 +0000 UTC m=+2925.159928538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8dffeea0-7e54-4c87-ba16-d3450b6403c2-provider-credentials") pod "payload-processing-external-model-79c9fb4bbb-6vx9l" (UID: "8dffeea0-7e54-4c87-ba16-d3450b6403c2") : secret "agc-cred-0e3c0a7229d2" not found
Oct 30 01:30:12.690094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:12.685772    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l"]
Oct 30 01:30:12.854095 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:12.853686    4864 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0e3c0a7229d2: secret "agc-cred-0e3c0a7229d2" not found
Oct 30 01:30:12.854095 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:12.853719    4864 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l: secret "agc-cred-0e3c0a7229d2" not found
Oct 30 01:30:12.854095 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:12.853791    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dffeea0-7e54-4c87-ba16-d3450b6403c2-provider-credentials podName:8dffeea0-7e54-4c87-ba16-d3450b6403c2 nodeName:}" failed. No retries permitted until 2026-10-30 01:30:13.853770361 +0000 UTC m=+2926.191872257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8dffeea0-7e54-4c87-ba16-d3450b6403c2-provider-credentials") pod "payload-processing-external-model-79c9fb4bbb-6vx9l" (UID: "8dffeea0-7e54-4c87-ba16-d3450b6403c2") : secret "agc-cred-0e3c0a7229d2" not found
Oct 30 01:30:13.872938 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:13.872897    4864 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0e3c0a7229d2: secret "agc-cred-0e3c0a7229d2" not found
Oct 30 01:30:13.872938 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:13.872928    4864 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l: secret "agc-cred-0e3c0a7229d2" not found
Oct 30 01:30:13.873378 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:13.872985    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dffeea0-7e54-4c87-ba16-d3450b6403c2-provider-credentials podName:8dffeea0-7e54-4c87-ba16-d3450b6403c2 nodeName:}" failed. No retries permitted until 2026-10-30 01:30:15.872968119 +0000 UTC m=+2928.211070032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8dffeea0-7e54-4c87-ba16-d3450b6403c2-provider-credentials") pod "payload-processing-external-model-79c9fb4bbb-6vx9l" (UID: "8dffeea0-7e54-4c87-ba16-d3450b6403c2") : secret "agc-cred-0e3c0a7229d2" not found
Oct 30 01:30:15.903903 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:15.903872    4864 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0e3c0a7229d2: secret "agc-cred-0e3c0a7229d2" not found
Oct 30 01:30:15.903903 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:15.903898    4864 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l: secret "agc-cred-0e3c0a7229d2" not found
Oct 30 01:30:15.904444 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:15.903961    4864 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dffeea0-7e54-4c87-ba16-d3450b6403c2-provider-credentials podName:8dffeea0-7e54-4c87-ba16-d3450b6403c2 nodeName:}" failed. No retries permitted until 2026-10-30 01:30:19.903943361 +0000 UTC m=+2932.242045264 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8dffeea0-7e54-4c87-ba16-d3450b6403c2-provider-credentials") pod "payload-processing-external-model-79c9fb4bbb-6vx9l" (UID: "8dffeea0-7e54-4c87-ba16-d3450b6403c2") : secret "agc-cred-0e3c0a7229d2" not found
Oct 30 01:30:16.663338 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.663301    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"]
Oct 30 01:30:16.668839 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.668820    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.686389 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.686298    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-499482-openshift-default-dockercfg-6fv68\""
Oct 30 01:30:16.701844 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.701814    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"]
Oct 30 01:30:16.816972 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.816938    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-workload-socket\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.817181 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.817159    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5krk\" (UniqueName: \"kubernetes.io/projected/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-kube-api-access-g5krk\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.817256 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.817197    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-data\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.817256 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.817229    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-envoy\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.817408 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.817286    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-workload-certs\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.817408 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.817371    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-token\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.817485 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.817441    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-podinfo\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.817515 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.817492    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istiod-ca-cert\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.817515 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.817511    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-credential-socket\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.918646 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.918558    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5krk\" (UniqueName: \"kubernetes.io/projected/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-kube-api-access-g5krk\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.918646 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.918618    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-data\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.919168 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.918662    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-envoy\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.919168 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.918687    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-workload-certs\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.919168 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.918733    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-token\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.919168 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.918805    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-podinfo\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.919168 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.918847    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istiod-ca-cert\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.919168 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.918873    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-credential-socket\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.919168 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.919022    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-workload-socket\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.930097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.921289    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-credential-socket\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.930097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.922110    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-workload-socket\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.932938 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.932907    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-workload-certs\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.933479 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.933421    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istiod-ca-cert\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.933621 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.933538    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-envoy\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.933946 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.933899    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-podinfo\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.934050 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.934032    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-data\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.945540 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.945517    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-token\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.956729 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.956700    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5krk\" (UniqueName: \"kubernetes.io/projected/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-kube-api-access-g5krk\") pod \"e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") " pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:16.983824 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:16.983792    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:17.322322 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:17.322297    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"]
Oct 30 01:30:17.323783 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:30:17.323751    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa04251_2153_46ef_b2fe_0b3583cfbaf9.slice/crio-9e8bc9f3f3202be144a543c13a7b807a6c4a44d0641d8883f3a6b0511d7460eb WatchSource:0}: Error finding container 9e8bc9f3f3202be144a543c13a7b807a6c4a44d0641d8883f3a6b0511d7460eb: Status 404 returned error can't find the container with id 9e8bc9f3f3202be144a543c13a7b807a6c4a44d0641d8883f3a6b0511d7460eb
Oct 30 01:30:17.327465 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:17.327397    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:30:17.327598 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:17.327569    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:30:17.327676 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:17.327658    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:30:18.222337 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:18.222300    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6" event={"ID":"2fa04251-2153-46ef-b2fe-0b3583cfbaf9","Type":"ContainerStarted","Data":"a2a4908f4da3a6c0745571feb1b04badfe0f22bf9e22229756bffde82389e770"}
Oct 30 01:30:18.222813 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:18.222370    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6" event={"ID":"2fa04251-2153-46ef-b2fe-0b3583cfbaf9","Type":"ContainerStarted","Data":"9e8bc9f3f3202be144a543c13a7b807a6c4a44d0641d8883f3a6b0511d7460eb"}
Oct 30 01:30:18.285092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:18.284062    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6" podStartSLOduration=2.284040121 podStartE2EDuration="2.284040121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:18.281199267 +0000 UTC m=+2930.619301215" watchObservedRunningTime="2026-10-30 01:30:18.284040121 +0000 UTC m=+2930.622142034"
Oct 30 01:30:18.787311 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:18.787208    4864 scope.go:122] "RemoveContainer" containerID="8a42c6b0e3c375eb724b803e25fb011a6093810c3856b7a287830eb2139e20a7"
Oct 30 01:30:18.787605 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:18.787581    4864 scope.go:122] "RemoveContainer" containerID="90078b4c869f67250d72c32fbf181eec5f4395137e3546b974be3cc4c340a33d"
Oct 30 01:30:18.986123 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:18.985226    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:18.986961 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:18.986925    4864 patch_prober.go:29] interesting pod/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 01:30:18.987102 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:18.987007    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6" podUID="2fa04251-2153-46ef-b2fe-0b3583cfbaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 01:30:19.155134 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:19.153818    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:30:19.182379 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:19.182148    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l" podUID="8dffeea0-7e54-4c87-ba16-d3450b6403c2" containerName="payload-processing" containerID="cri-o://f709e7c51a4c31ed8b1b8e4901ed6e2b2a4cb48e4c86088ecab1134462ec2517" gracePeriod=25
Oct 30 01:30:19.231084 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:19.231042    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn" event={"ID":"abc4e617-7562-4f35-b084-aa303323866b","Type":"ContainerStarted","Data":"00faea600dd9a93114854435ecba51705afca6a72ca5ef920c4c9b22be88fac4"}
Oct 30 01:30:19.287871 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:19.287814    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn" podStartSLOduration=14.76065184 podStartE2EDuration="19.28779366s" totalImagesPullingTime="4.52714182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:19.268476405 +0000 UTC m=+2931.606578340" watchObservedRunningTime="2026-10-30 01:30:19.28779366 +0000 UTC m=+2931.625895570"
Oct 30 01:30:19.585124 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:19.578298    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l"
Oct 30 01:30:19.626260 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:19.626207    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dffeea0-7e54-4c87-ba16-d3450b6403c2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8dffeea0-7e54-4c87-ba16-d3450b6403c2-routing-overlay-volume\") pod \"8dffeea0-7e54-4c87-ba16-d3450b6403c2\" (UID: \"8dffeea0-7e54-4c87-ba16-d3450b6403c2\") "
Oct 30 01:30:19.626453 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:19.626312    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dffeea0-7e54-4c87-ba16-d3450b6403c2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8dffeea0-7e54-4c87-ba16-d3450b6403c2-provider-credentials\") pod \"8dffeea0-7e54-4c87-ba16-d3450b6403c2\" (UID: \"8dffeea0-7e54-4c87-ba16-d3450b6403c2\") "
Oct 30 01:30:19.626453 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:19.626427    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dffeea0-7e54-4c87-ba16-d3450b6403c2-kube-api-access-xszpz\" (UniqueName: \"kubernetes.io/projected/8dffeea0-7e54-4c87-ba16-d3450b6403c2-kube-api-access-xszpz\") pod \"8dffeea0-7e54-4c87-ba16-d3450b6403c2\" (UID: \"8dffeea0-7e54-4c87-ba16-d3450b6403c2\") "
Oct 30 01:30:19.626699 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:19.626674    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dffeea0-7e54-4c87-ba16-d3450b6403c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dffeea0-7e54-4c87-ba16-d3450b6403c2-plugins-config-volume\") pod \"8dffeea0-7e54-4c87-ba16-d3450b6403c2\" (UID: \"8dffeea0-7e54-4c87-ba16-d3450b6403c2\") "
Oct 30 01:30:19.630419 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:19.630381    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dffeea0-7e54-4c87-ba16-d3450b6403c2-plugins-config-volume" pod "8dffeea0-7e54-4c87-ba16-d3450b6403c2" (UID: "8dffeea0-7e54-4c87-ba16-d3450b6403c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:19.631124 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:19.630774    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dffeea0-7e54-4c87-ba16-d3450b6403c2-routing-overlay-volume" pod "8dffeea0-7e54-4c87-ba16-d3450b6403c2" (UID: "8dffeea0-7e54-4c87-ba16-d3450b6403c2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:19.655934 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:19.653552    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dffeea0-7e54-4c87-ba16-d3450b6403c2-provider-credentials" pod "8dffeea0-7e54-4c87-ba16-d3450b6403c2" (UID: "8dffeea0-7e54-4c87-ba16-d3450b6403c2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:19.809378 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:19.793341    4864 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dffeea0-7e54-4c87-ba16-d3450b6403c2-plugins-config-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:30:19.809378 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:19.793376    4864 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8dffeea0-7e54-4c87-ba16-d3450b6403c2-routing-overlay-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:30:19.809378 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:19.793393    4864 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8dffeea0-7e54-4c87-ba16-d3450b6403c2-provider-credentials\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:30:19.879216 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:19.879179    4864 scope.go:122] "RemoveContainer" containerID="4dc150c259098b944953b8e20ad2b3b00485da810c0610cc81d7d46f1c0be984"
Oct 30 01:30:19.910453 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:19.907480    4864 scope.go:122] "RemoveContainer" containerID="327ac44aab278ddb38f0d3bb2b51f640483653afbbb0cff6e4e45da14788dd1b"
Oct 30 01:30:19.947247 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:19.947209    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dffeea0-7e54-4c87-ba16-d3450b6403c2-kube-api-access-xszpz" pod "8dffeea0-7e54-4c87-ba16-d3450b6403c2" (UID: "8dffeea0-7e54-4c87-ba16-d3450b6403c2"). InnerVolumeSpecName "kube-api-access-xszpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:19.953870 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:19.953852    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"]
Oct 30 01:30:19.954684 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:19.954666    4864 cpu_manager.go:414] "containerMap: removing container" podUID="8dffeea0-7e54-4c87-ba16-d3450b6403c2" containerName="payload-processing"
Oct 30 01:30:19.954780 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:19.954771    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dffeea0-7e54-4c87-ba16-d3450b6403c2" containerName="payload-processing"
Oct 30 01:30:19.954969 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:19.954961    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dffeea0-7e54-4c87-ba16-d3450b6403c2" containerName="payload-processing"
Oct 30 01:30:19.990457 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:19.989486    4864 patch_prober.go:29] interesting pod/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 01:30:19.990457 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:19.989537    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6" podUID="2fa04251-2153-46ef-b2fe-0b3583cfbaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 01:30:19.991657 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:19.991636    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.003279 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.003248    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xszpz\" (UniqueName: \"kubernetes.io/projected/8dffeea0-7e54-4c87-ba16-d3450b6403c2-kube-api-access-xszpz\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:30:20.010180 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.010145    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"]
Oct 30 01:30:20.014651 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.014628    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.017037 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.017017    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"]
Oct 30 01:30:20.024579 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.024550    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"]
Oct 30 01:30:20.104553 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.104517    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.104685 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.104615    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.104685 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.104647    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.104761 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.104728    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.104793 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.104781    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwl52\" (UniqueName: \"kubernetes.io/projected/0d79aa0b-3be0-4837-8182-e1e940a4d73a-kube-api-access-dwl52\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.104985 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.104897    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.105168 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.105143    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.105247 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.105179    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.105314 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.105284    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.206949 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.206916    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.207150 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.206957    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.207150 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.207007    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01101e66-e714-4a99-af87-3e8e0e895867-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.207150 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.207061    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.207150 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.207137    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.207408 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.207175    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.207408 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.207248    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.207408 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.207326    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.207408 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.207386    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.207634 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.207608    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.207745 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.207652    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01101e66-e714-4a99-af87-3e8e0e895867-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.207798 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.207764    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.208377 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.208354    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.209223 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.208994    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.212619 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.212595    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.213379 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.213354    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.213563 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.213541    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.213707 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.213686    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwl52\" (UniqueName: \"kubernetes.io/projected/0d79aa0b-3be0-4837-8182-e1e940a4d73a-kube-api-access-dwl52\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.213856 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.213839    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01101e66-e714-4a99-af87-3e8e0e895867-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.214015 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.213996    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfnhq\" (UniqueName: \"kubernetes.io/projected/01101e66-e714-4a99-af87-3e8e0e895867-kube-api-access-rfnhq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.214201 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.214180    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.214389 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.214363    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.214589 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.214459    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.240446 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.240409    4864 generic.go:356] "Generic (PLEG): container finished" podID="8dffeea0-7e54-4c87-ba16-d3450b6403c2" containerID="f709e7c51a4c31ed8b1b8e4901ed6e2b2a4cb48e4c86088ecab1134462ec2517" exitCode=0
Oct 30 01:30:20.240788 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.240599    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l" event={"ID":"8dffeea0-7e54-4c87-ba16-d3450b6403c2","Type":"ContainerDied","Data":"f709e7c51a4c31ed8b1b8e4901ed6e2b2a4cb48e4c86088ecab1134462ec2517"}
Oct 30 01:30:20.240788 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.240689    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l" event={"ID":"8dffeea0-7e54-4c87-ba16-d3450b6403c2","Type":"ContainerDied","Data":"a740122fc1de5ee64000788bc48826b90df55aca7f7d416366f0871370aeb9da"}
Oct 30 01:30:20.240788 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.240721    4864 scope.go:122] "RemoveContainer" containerID="f709e7c51a4c31ed8b1b8e4901ed6e2b2a4cb48e4c86088ecab1134462ec2517"
Oct 30 01:30:20.240898 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.240885    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l"
Oct 30 01:30:20.249300 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.249264    4864 generic.go:356] "Generic (PLEG): container finished" podID="abc4e617-7562-4f35-b084-aa303323866b" containerID="00faea600dd9a93114854435ecba51705afca6a72ca5ef920c4c9b22be88fac4" exitCode=60
Oct 30 01:30:20.249419 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.249332    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn" event={"ID":"abc4e617-7562-4f35-b084-aa303323866b","Type":"ContainerDied","Data":"00faea600dd9a93114854435ecba51705afca6a72ca5ef920c4c9b22be88fac4"}
Oct 30 01:30:20.249920 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.249895    4864 scope.go:122] "RemoveContainer" containerID="00faea600dd9a93114854435ecba51705afca6a72ca5ef920c4c9b22be88fac4"
Oct 30 01:30:20.250344 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:20.250311    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn_odh-ai-gateway-infra(abc4e617-7562-4f35-b084-aa303323866b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn" podUID="abc4e617-7562-4f35-b084-aa303323866b"
Oct 30 01:30:20.255303 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.252925    4864 generic.go:356] "Generic (PLEG): container finished" podID="94b230f0-95fe-42ce-8fdf-8fea818268e3" containerID="bf7b4bc2ff6e05e8ac010bd01782091dbc34a043a61cf669ad70844f1e2ab4d9" exitCode=60
Oct 30 01:30:20.255303 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.253141    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx" event={"ID":"94b230f0-95fe-42ce-8fdf-8fea818268e3","Type":"ContainerDied","Data":"bf7b4bc2ff6e05e8ac010bd01782091dbc34a043a61cf669ad70844f1e2ab4d9"}
Oct 30 01:30:20.255303 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.253629    4864 scope.go:122] "RemoveContainer" containerID="bf7b4bc2ff6e05e8ac010bd01782091dbc34a043a61cf669ad70844f1e2ab4d9"
Oct 30 01:30:20.255303 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:20.253925    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx_odh-ai-gateway-infra(94b230f0-95fe-42ce-8fdf-8fea818268e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx" podUID="94b230f0-95fe-42ce-8fdf-8fea818268e3"
Oct 30 01:30:20.285707 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.285671    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.287196 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.287169    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.288478 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.288455    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwl52\" (UniqueName: \"kubernetes.io/projected/0d79aa0b-3be0-4837-8182-e1e940a4d73a-kube-api-access-dwl52\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.291600 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.291573    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zz9kn\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.292347 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.292323    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn"]
Oct 30 01:30:20.297396 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.297275    4864 scope.go:122] "RemoveContainer" containerID="f709e7c51a4c31ed8b1b8e4901ed6e2b2a4cb48e4c86088ecab1134462ec2517"
Oct 30 01:30:20.297891 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:20.297777    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f709e7c51a4c31ed8b1b8e4901ed6e2b2a4cb48e4c86088ecab1134462ec2517\": container with ID starting with f709e7c51a4c31ed8b1b8e4901ed6e2b2a4cb48e4c86088ecab1134462ec2517 not found: ID does not exist" containerID="f709e7c51a4c31ed8b1b8e4901ed6e2b2a4cb48e4c86088ecab1134462ec2517"
Oct 30 01:30:20.297891 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.297801    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f709e7c51a4c31ed8b1b8e4901ed6e2b2a4cb48e4c86088ecab1134462ec2517"} err="failed to get container status \"f709e7c51a4c31ed8b1b8e4901ed6e2b2a4cb48e4c86088ecab1134462ec2517\": rpc error: code = NotFound desc = could not find container \"f709e7c51a4c31ed8b1b8e4901ed6e2b2a4cb48e4c86088ecab1134462ec2517\": container with ID starting with f709e7c51a4c31ed8b1b8e4901ed6e2b2a4cb48e4c86088ecab1134462ec2517 not found: ID does not exist"
Oct 30 01:30:20.297891 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.297820    4864 scope.go:122] "RemoveContainer" containerID="8a42c6b0e3c375eb724b803e25fb011a6093810c3856b7a287830eb2139e20a7"
Oct 30 01:30:20.316951 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.316926    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01101e66-e714-4a99-af87-3e8e0e895867-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.318814 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.318433    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01101e66-e714-4a99-af87-3e8e0e895867-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.318814 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.318447    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.318814 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.318522    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.318814 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.318609    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01101e66-e714-4a99-af87-3e8e0e895867-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.318814 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.318772    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.319199 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.318845    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01101e66-e714-4a99-af87-3e8e0e895867-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.319199 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.318929    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfnhq\" (UniqueName: \"kubernetes.io/projected/01101e66-e714-4a99-af87-3e8e0e895867-kube-api-access-rfnhq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.319199 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.319136    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.319365 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.319281    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.320900 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.319647    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.320900 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.319698    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.320900 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.319952    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.321129 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.321021    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.323520 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.323500    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.323803 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.323784    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01101e66-e714-4a99-af87-3e8e0e895867-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.351092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.351052    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfnhq\" (UniqueName: \"kubernetes.io/projected/01101e66-e714-4a99-af87-3e8e0e895867-kube-api-access-rfnhq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.353632 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.353605    4864 scope.go:122] "RemoveContainer" containerID="90078b4c869f67250d72c32fbf181eec5f4395137e3546b974be3cc4c340a33d"
Oct 30 01:30:20.360093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.356276    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01101e66-e714-4a99-af87-3e8e0e895867-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2h99j\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.423645 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.423603    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:20.430939 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.430905    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:20.431096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.431064    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l"]
Oct 30 01:30:20.441572 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.441545    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-79c9fb4bbb-6vx9l"]
Oct 30 01:30:20.792693 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.792669    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"]
Oct 30 01:30:20.797386 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:30:20.797356    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d79aa0b_3be0_4837_8182_e1e940a4d73a.slice/crio-5e5adca015c5985235e9c10839fb791ff391c72c10ba51f5a28ea8af0ef6742b WatchSource:0}: Error finding container 5e5adca015c5985235e9c10839fb791ff391c72c10ba51f5a28ea8af0ef6742b: Status 404 returned error can't find the container with id 5e5adca015c5985235e9c10839fb791ff391c72c10ba51f5a28ea8af0ef6742b
Oct 30 01:30:20.801599 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.801519    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:30:20.801670 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.801605    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:30:20.801670 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.801647    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:30:20.802615 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.802592    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"]
Oct 30 01:30:20.804347 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:30:20.804319    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01101e66_e714_4a99_af87_3e8e0e895867.slice/crio-fa3441cf72c27a74dc8ee10e6bbfc236ff823e80d9c8320c8aa570f132b5da44 WatchSource:0}: Error finding container fa3441cf72c27a74dc8ee10e6bbfc236ff823e80d9c8320c8aa570f132b5da44: Status 404 returned error can't find the container with id fa3441cf72c27a74dc8ee10e6bbfc236ff823e80d9c8320c8aa570f132b5da44
Oct 30 01:30:20.815007 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.814969    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:30:20.815130 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.815060    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:30:20.815199 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.815155    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:30:20.990946 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.990909    4864 patch_prober.go:29] interesting pod/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 01:30:20.991214 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:20.991181    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6" podUID="2fa04251-2153-46ef-b2fe-0b3583cfbaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 01:30:21.321903 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.321813    4864 generic.go:356] "Generic (PLEG): container finished" podID="56d653be-f408-4827-b79a-fba3572cc69d" containerID="5281e7501c6b618cf6e6618d4eeb843cb3918d970cbe4f0d703ed80bee36a7f0" exitCode=60
Oct 30 01:30:21.323678 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.322129    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg" event={"ID":"56d653be-f408-4827-b79a-fba3572cc69d","Type":"ContainerDied","Data":"5281e7501c6b618cf6e6618d4eeb843cb3918d970cbe4f0d703ed80bee36a7f0"}
Oct 30 01:30:21.323678 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.322499    4864 scope.go:122] "RemoveContainer" containerID="327ac44aab278ddb38f0d3bb2b51f640483653afbbb0cff6e4e45da14788dd1b"
Oct 30 01:30:21.323678 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.322954    4864 scope.go:122] "RemoveContainer" containerID="5281e7501c6b618cf6e6618d4eeb843cb3918d970cbe4f0d703ed80bee36a7f0"
Oct 30 01:30:21.323678 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:21.323378    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg_odh-ai-gateway-infra(56d653be-f408-4827-b79a-fba3572cc69d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg" podUID="56d653be-f408-4827-b79a-fba3572cc69d"
Oct 30 01:30:21.329568 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.329502    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j" event={"ID":"01101e66-e714-4a99-af87-3e8e0e895867","Type":"ContainerStarted","Data":"804d68c1125a5defc4e3db4f0c20fe1f40192d451cd0e62af663dc305ff9bfba"}
Oct 30 01:30:21.329680 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.329591    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j" event={"ID":"01101e66-e714-4a99-af87-3e8e0e895867","Type":"ContainerStarted","Data":"fa3441cf72c27a74dc8ee10e6bbfc236ff823e80d9c8320c8aa570f132b5da44"}
Oct 30 01:30:21.335562 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.334122    4864 generic.go:356] "Generic (PLEG): container finished" podID="40755b18-80e6-415f-8a27-4879ca6222b8" containerID="76a10130c098b71639a376d26919762ddf54cb6011d380f24ab63e4c71761098" exitCode=60
Oct 30 01:30:21.335562 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.334205    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9" event={"ID":"40755b18-80e6-415f-8a27-4879ca6222b8","Type":"ContainerDied","Data":"76a10130c098b71639a376d26919762ddf54cb6011d380f24ab63e4c71761098"}
Oct 30 01:30:21.335562 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.334596    4864 scope.go:122] "RemoveContainer" containerID="76a10130c098b71639a376d26919762ddf54cb6011d380f24ab63e4c71761098"
Oct 30 01:30:21.335562 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:21.334959    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9_odh-ai-gateway-infra(40755b18-80e6-415f-8a27-4879ca6222b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9" podUID="40755b18-80e6-415f-8a27-4879ca6222b8"
Oct 30 01:30:21.338952 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.338933    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn" event={"ID":"0d79aa0b-3be0-4837-8182-e1e940a4d73a","Type":"ContainerStarted","Data":"3b76fcda9bc226478d5db559ad4e6233435ea901203dc71040112a0c7b5d29bf"}
Oct 30 01:30:21.339146 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.339127    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn" event={"ID":"0d79aa0b-3be0-4837-8182-e1e940a4d73a","Type":"ContainerStarted","Data":"5e5adca015c5985235e9c10839fb791ff391c72c10ba51f5a28ea8af0ef6742b"}
Oct 30 01:30:21.344565 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.344545    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx"]
Oct 30 01:30:21.429526 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.429489    4864 scope.go:122] "RemoveContainer" containerID="4dc150c259098b944953b8e20ad2b3b00485da810c0610cc81d7d46f1c0be984"
Oct 30 01:30:21.429698 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.429615    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:21.430115 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.430086    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zz9kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 01:30:21.430208 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.430141    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn" podUID="0d79aa0b-3be0-4837-8182-e1e940a4d73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 01:30:21.431163 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.431140    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:21.432422 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.432398    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2h99j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 01:30:21.432576 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.432444    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j" podUID="01101e66-e714-4a99-af87-3e8e0e895867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 01:30:21.496234 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.475469    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn" podStartSLOduration=2.475429658 podStartE2EDuration="2.475429658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:21.467405451 +0000 UTC m=+2933.805507377" watchObservedRunningTime="2026-10-30 01:30:21.475429658 +0000 UTC m=+2933.813531595"
Oct 30 01:30:21.666230 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.657848    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j" podStartSLOduration=2.657825422 podStartE2EDuration="2.657825422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:21.654300284 +0000 UTC m=+2933.992402194" watchObservedRunningTime="2026-10-30 01:30:21.657825422 +0000 UTC m=+2933.995927333"
Oct 30 01:30:21.784415 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.784384    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn"
Oct 30 01:30:21.793002 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.792977    4864 scope.go:122] "RemoveContainer" containerID="8ac5c54282b70219ca162a1546bea1ed5cd52b304e43e2e84a62a28f66397e58"
Oct 30 01:30:21.832483 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.832445    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dffeea0-7e54-4c87-ba16-d3450b6403c2" path="/var/lib/kubelet/pods/8dffeea0-7e54-4c87-ba16-d3450b6403c2/volumes"
Oct 30 01:30:21.843432 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.843400    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc4e617-7562-4f35-b084-aa303323866b-kube-api-access-4xl52\" (UniqueName: \"kubernetes.io/projected/abc4e617-7562-4f35-b084-aa303323866b-kube-api-access-4xl52\") pod \"abc4e617-7562-4f35-b084-aa303323866b\" (UID: \"abc4e617-7562-4f35-b084-aa303323866b\") "
Oct 30 01:30:21.843685 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.843666    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc4e617-7562-4f35-b084-aa303323866b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc4e617-7562-4f35-b084-aa303323866b-maas-api-service-ca\") pod \"abc4e617-7562-4f35-b084-aa303323866b\" (UID: \"abc4e617-7562-4f35-b084-aa303323866b\") "
Oct 30 01:30:21.846266 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.846235    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc4e617-7562-4f35-b084-aa303323866b-maas-api-service-ca" pod "abc4e617-7562-4f35-b084-aa303323866b" (UID: "abc4e617-7562-4f35-b084-aa303323866b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:21.855695 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.855662    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc4e617-7562-4f35-b084-aa303323866b-kube-api-access-4xl52" pod "abc4e617-7562-4f35-b084-aa303323866b" (UID: "abc4e617-7562-4f35-b084-aa303323866b"). InnerVolumeSpecName "kube-api-access-4xl52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:21.903101 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.903011    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx"
Oct 30 01:30:21.945901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.945868    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94b230f0-95fe-42ce-8fdf-8fea818268e3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94b230f0-95fe-42ce-8fdf-8fea818268e3-maas-api-service-ca\") pod \"94b230f0-95fe-42ce-8fdf-8fea818268e3\" (UID: \"94b230f0-95fe-42ce-8fdf-8fea818268e3\") "
Oct 30 01:30:21.946331 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.946308    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94b230f0-95fe-42ce-8fdf-8fea818268e3-kube-api-access-f52vs\" (UniqueName: \"kubernetes.io/projected/94b230f0-95fe-42ce-8fdf-8fea818268e3-kube-api-access-f52vs\") pod \"94b230f0-95fe-42ce-8fdf-8fea818268e3\" (UID: \"94b230f0-95fe-42ce-8fdf-8fea818268e3\") "
Oct 30 01:30:21.947066 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.946472    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b230f0-95fe-42ce-8fdf-8fea818268e3-maas-api-service-ca" pod "94b230f0-95fe-42ce-8fdf-8fea818268e3" (UID: "94b230f0-95fe-42ce-8fdf-8fea818268e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:21.947526 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.947513    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc4e617-7562-4f35-b084-aa303323866b-maas-api-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:30:21.947707 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.947689    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xl52\" (UniqueName: \"kubernetes.io/projected/abc4e617-7562-4f35-b084-aa303323866b-kube-api-access-4xl52\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:30:21.947810 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.947797    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94b230f0-95fe-42ce-8fdf-8fea818268e3-maas-api-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:30:21.949015 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.948984    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b230f0-95fe-42ce-8fdf-8fea818268e3-kube-api-access-f52vs" pod "94b230f0-95fe-42ce-8fdf-8fea818268e3" (UID: "94b230f0-95fe-42ce-8fdf-8fea818268e3"). InnerVolumeSpecName "kube-api-access-f52vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:21.987538 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.987497    4864 patch_prober.go:29] interesting pod/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 01:30:21.988065 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:21.988033    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6" podUID="2fa04251-2153-46ef-b2fe-0b3583cfbaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 01:30:22.049481 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:22.049452    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f52vs\" (UniqueName: \"kubernetes.io/projected/94b230f0-95fe-42ce-8fdf-8fea818268e3-kube-api-access-f52vs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:30:22.409098 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:22.396851    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn" event={"ID":"abc4e617-7562-4f35-b084-aa303323866b","Type":"ContainerDied","Data":"f1cd5b88c643c5b7e2d829a85ccc0bb63190c5fb102ce27e2a8658697b0ae476"}
Oct 30 01:30:22.409098 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:22.396957    4864 scope.go:122] "RemoveContainer" containerID="00faea600dd9a93114854435ecba51705afca6a72ca5ef920c4c9b22be88fac4"
Oct 30 01:30:22.409098 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:22.397186    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn"
Oct 30 01:30:22.424753 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:22.424305    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zz9kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 01:30:22.424753 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:22.424376    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn" podUID="0d79aa0b-3be0-4837-8182-e1e940a4d73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 01:30:22.426936 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:22.426893    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg"]
Oct 30 01:30:22.435526 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:22.434604    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2h99j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 01:30:22.435526 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:22.434640    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j" podUID="01101e66-e714-4a99-af87-3e8e0e895867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 01:30:22.435526 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:22.435234    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx"
Oct 30 01:30:22.435885 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:22.435792    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx" event={"ID":"94b230f0-95fe-42ce-8fdf-8fea818268e3","Type":"ContainerDied","Data":"68087f3294a639a918f81728d3f759f7965d0169ec4b0fc32caeb2b36edf5775"}
Oct 30 01:30:22.447209 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:22.447168    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9"]
Oct 30 01:30:22.501230 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:22.501130    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn"]
Oct 30 01:30:22.557012 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:22.556979    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888730-n4xhn"]
Oct 30 01:30:22.613702 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:22.613589    4864 scope.go:122] "RemoveContainer" containerID="bf7b4bc2ff6e05e8ac010bd01782091dbc34a043a61cf669ad70844f1e2ab4d9"
Oct 30 01:30:22.790111 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:22.776116    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx"]
Oct 30 01:30:22.803114 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:22.799420    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888730-dbwwx"]
Oct 30 01:30:23.051144 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:23.050762    4864 patch_prober.go:29] interesting pod/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 01:30:23.051144 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:23.050831    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6" podUID="2fa04251-2153-46ef-b2fe-0b3583cfbaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 01:30:23.270621 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:23.270304    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg"
Oct 30 01:30:23.379553 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:23.379463    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d653be-f408-4827-b79a-fba3572cc69d-kube-api-access-xl42s\" (UniqueName: \"kubernetes.io/projected/56d653be-f408-4827-b79a-fba3572cc69d-kube-api-access-xl42s\") pod \"56d653be-f408-4827-b79a-fba3572cc69d\" (UID: \"56d653be-f408-4827-b79a-fba3572cc69d\") "
Oct 30 01:30:23.380064 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:23.380030    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56d653be-f408-4827-b79a-fba3572cc69d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d653be-f408-4827-b79a-fba3572cc69d-maas-api-service-ca\") pod \"56d653be-f408-4827-b79a-fba3572cc69d\" (UID: \"56d653be-f408-4827-b79a-fba3572cc69d\") "
Oct 30 01:30:23.435427 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:23.414372    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d653be-f408-4827-b79a-fba3572cc69d-maas-api-service-ca" pod "56d653be-f408-4827-b79a-fba3572cc69d" (UID: "56d653be-f408-4827-b79a-fba3572cc69d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:23.460622 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:23.460582    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d653be-f408-4827-b79a-fba3572cc69d-kube-api-access-xl42s" pod "56d653be-f408-4827-b79a-fba3572cc69d" (UID: "56d653be-f408-4827-b79a-fba3572cc69d"). InnerVolumeSpecName "kube-api-access-xl42s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:23.509093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:23.505325    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d653be-f408-4827-b79a-fba3572cc69d-maas-api-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:30:23.509093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:23.505360    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl42s\" (UniqueName: \"kubernetes.io/projected/56d653be-f408-4827-b79a-fba3572cc69d-kube-api-access-xl42s\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:30:23.652142 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:23.645621    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2h99j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 01:30:23.652142 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:23.645683    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j" podUID="01101e66-e714-4a99-af87-3e8e0e895867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 01:30:23.812660 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:23.802428    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zz9kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 01:30:23.812660 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:23.802492    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn" podUID="0d79aa0b-3be0-4837-8182-e1e940a4d73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 01:30:23.927936 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:23.927282    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj" podUID="63a8525d-9951-4f19-b124-62e4fa91c6b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:30:23.969105 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:23.956497    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll" podUID="2388a847-21ff-46dd-8b45-4183a2448cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:30:24.055092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.039867    4864 patch_prober.go:29] interesting pod/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 01:30:24.055092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.039927    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6" podUID="2fa04251-2153-46ef-b2fe-0b3583cfbaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 01:30:24.060203 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.060172    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg"
Oct 30 01:30:24.311126 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.289840    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9"
Oct 30 01:30:24.311126 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.290804    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94b230f0-95fe-42ce-8fdf-8fea818268e3" path="/var/lib/kubelet/pods/94b230f0-95fe-42ce-8fdf-8fea818268e3/volumes"
Oct 30 01:30:24.311126 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.291733    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc4e617-7562-4f35-b084-aa303323866b" path="/var/lib/kubelet/pods/abc4e617-7562-4f35-b084-aa303323866b/volumes"
Oct 30 01:30:24.311126 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.292500    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg" event={"ID":"56d653be-f408-4827-b79a-fba3572cc69d","Type":"ContainerDied","Data":"c03886758b5fb3a5dd757939b115bea5a644424ac2db4351d8373b4d2cde9cd4"}
Oct 30 01:30:24.311126 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.292585    4864 scope.go:122] "RemoveContainer" containerID="5281e7501c6b618cf6e6618d4eeb843cb3918d970cbe4f0d703ed80bee36a7f0"
Oct 30 01:30:24.383159 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.343417    4864 generic.go:356] "Generic (PLEG): container finished" podID="ce2530ca-6609-45c1-8438-17393a09e6c9" containerID="d709438ac3fe804ba3a5d53c852336acab363889a4cd13143bb25abc5e260af5" exitCode=60
Oct 30 01:30:24.383159 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.343676    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6" event={"ID":"ce2530ca-6609-45c1-8438-17393a09e6c9","Type":"ContainerDied","Data":"d709438ac3fe804ba3a5d53c852336acab363889a4cd13143bb25abc5e260af5"}
Oct 30 01:30:24.383159 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.364273    4864 scope.go:122] "RemoveContainer" containerID="d709438ac3fe804ba3a5d53c852336acab363889a4cd13143bb25abc5e260af5"
Oct 30 01:30:24.383159 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:24.364711    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6_odh-ai-gateway-infra(ce2530ca-6609-45c1-8438-17393a09e6c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6" podUID="ce2530ca-6609-45c1-8438-17393a09e6c9"
Oct 30 01:30:24.426798 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.426760    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40755b18-80e6-415f-8a27-4879ca6222b8-maas-api-service-ca" pod "40755b18-80e6-415f-8a27-4879ca6222b8" (UID: "40755b18-80e6-415f-8a27-4879ca6222b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:24.427391 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.427361    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zz9kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 01:30:24.427584 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.427557    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn" podUID="0d79aa0b-3be0-4837-8182-e1e940a4d73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 01:30:24.439445 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.439412    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2h99j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 01:30:24.439959 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.439927    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j" podUID="01101e66-e714-4a99-af87-3e8e0e895867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 01:30:24.445196 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.445147    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40755b18-80e6-415f-8a27-4879ca6222b8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40755b18-80e6-415f-8a27-4879ca6222b8-maas-api-service-ca\") pod \"40755b18-80e6-415f-8a27-4879ca6222b8\" (UID: \"40755b18-80e6-415f-8a27-4879ca6222b8\") "
Oct 30 01:30:24.445639 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.445615    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40755b18-80e6-415f-8a27-4879ca6222b8-kube-api-access-9t92g\" (UniqueName: \"kubernetes.io/projected/40755b18-80e6-415f-8a27-4879ca6222b8-kube-api-access-9t92g\") pod \"40755b18-80e6-415f-8a27-4879ca6222b8\" (UID: \"40755b18-80e6-415f-8a27-4879ca6222b8\") "
Oct 30 01:30:24.446979 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.446956    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40755b18-80e6-415f-8a27-4879ca6222b8-maas-api-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:30:24.504582 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.502338    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj" podUID="8b267884-baae-4ad3-a414-82467f7188b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:30:24.548831 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.548778    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40755b18-80e6-415f-8a27-4879ca6222b8-kube-api-access-9t92g" pod "40755b18-80e6-415f-8a27-4879ca6222b8" (UID: "40755b18-80e6-415f-8a27-4879ca6222b8"). InnerVolumeSpecName "kube-api-access-9t92g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:24.650911 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.650751    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t92g\" (UniqueName: \"kubernetes.io/projected/40755b18-80e6-415f-8a27-4879ca6222b8-kube-api-access-9t92g\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:30:24.726419 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.726203    4864 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll" podUID="2388a847-21ff-46dd-8b45-4183a2448cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:30:24.743411 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.743381    4864 scope.go:122] "RemoveContainer" containerID="8ac5c54282b70219ca162a1546bea1ed5cd52b304e43e2e84a62a28f66397e58"
Oct 30 01:30:24.750232 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.750194    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg"]
Oct 30 01:30:24.754452 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:24.754421    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888730-kf8fg"]
Oct 30 01:30:25.461015 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:25.454308    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2h99j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 01:30:25.461015 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:25.454379    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j" podUID="01101e66-e714-4a99-af87-3e8e0e895867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 01:30:25.461015 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:25.454890    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zz9kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 01:30:25.461015 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:25.454923    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn" podUID="0d79aa0b-3be0-4837-8182-e1e940a4d73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 01:30:25.497387 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:25.496128    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6"]
Oct 30 01:30:25.497387 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:25.497116    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:25.515120 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:25.510382    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:25.585178 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:25.574386    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:30:25.643804 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:25.643769    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9" event={"ID":"40755b18-80e6-415f-8a27-4879ca6222b8","Type":"ContainerDied","Data":"2845cf76f38065871d552d6bdb885d027ac0ef4776df071b72e3f125ede461f5"}
Oct 30 01:30:25.643924 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:25.643830    4864 scope.go:122] "RemoveContainer" containerID="76a10130c098b71639a376d26919762ddf54cb6011d380f24ab63e4c71761098"
Oct 30 01:30:25.643961 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:25.643955    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9"
Oct 30 01:30:25.864154 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:25.826439    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56d653be-f408-4827-b79a-fba3572cc69d" path="/var/lib/kubelet/pods/56d653be-f408-4827-b79a-fba3572cc69d/volumes"
Oct 30 01:30:25.939242 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:25.939209    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9"]
Oct 30 01:30:26.092096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:26.091190    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888730-vs2r9"]
Oct 30 01:30:26.424392 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:26.424361    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zz9kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 01:30:26.424530 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:26.424441    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn" podUID="0d79aa0b-3be0-4837-8182-e1e940a4d73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 01:30:26.425578 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:26.425561    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6"
Oct 30 01:30:26.431666 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:26.431646    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2h99j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 01:30:26.431726 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:26.431684    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j" podUID="01101e66-e714-4a99-af87-3e8e0e895867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 01:30:26.546263 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:26.546229    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce2530ca-6609-45c1-8438-17393a09e6c9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2530ca-6609-45c1-8438-17393a09e6c9-maas-api-service-ca\") pod \"ce2530ca-6609-45c1-8438-17393a09e6c9\" (UID: \"ce2530ca-6609-45c1-8438-17393a09e6c9\") "
Oct 30 01:30:26.546772 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:26.546425    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce2530ca-6609-45c1-8438-17393a09e6c9-kube-api-access-6wfsh\" (UniqueName: \"kubernetes.io/projected/ce2530ca-6609-45c1-8438-17393a09e6c9-kube-api-access-6wfsh\") pod \"ce2530ca-6609-45c1-8438-17393a09e6c9\" (UID: \"ce2530ca-6609-45c1-8438-17393a09e6c9\") "
Oct 30 01:30:26.546772 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:26.546638    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce2530ca-6609-45c1-8438-17393a09e6c9-maas-api-service-ca" pod "ce2530ca-6609-45c1-8438-17393a09e6c9" (UID: "ce2530ca-6609-45c1-8438-17393a09e6c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:26.547215 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:26.547175    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2530ca-6609-45c1-8438-17393a09e6c9-maas-api-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:30:26.549191 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:26.549161    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce2530ca-6609-45c1-8438-17393a09e6c9-kube-api-access-6wfsh" pod "ce2530ca-6609-45c1-8438-17393a09e6c9" (UID: "ce2530ca-6609-45c1-8438-17393a09e6c9"). InnerVolumeSpecName "kube-api-access-6wfsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:26.735095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:26.666346    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wfsh\" (UniqueName: \"kubernetes.io/projected/ce2530ca-6609-45c1-8438-17393a09e6c9-kube-api-access-6wfsh\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:30:26.761034 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:26.760943    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6" event={"ID":"ce2530ca-6609-45c1-8438-17393a09e6c9","Type":"ContainerDied","Data":"e0b526f8851170249ff0cd16044c0b382a61968345328eaaaa21533bcf5786f3"}
Oct 30 01:30:26.773207 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:26.773172    4864 scope.go:122] "RemoveContainer" containerID="d709438ac3fe804ba3a5d53c852336acab363889a4cd13143bb25abc5e260af5"
Oct 30 01:30:26.773489 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:26.767902    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6"
Oct 30 01:30:26.912526 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:26.912483    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6"]
Oct 30 01:30:26.916724 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:26.916694    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-499482-29888730-jp5x6"]
Oct 30 01:30:27.424925 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:27.424887    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zz9kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 01:30:27.425111 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:27.424941    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn" podUID="0d79aa0b-3be0-4837-8182-e1e940a4d73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 01:30:27.431996 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:27.431974    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2h99j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 01:30:27.432094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:27.432016    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j" podUID="01101e66-e714-4a99-af87-3e8e0e895867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 01:30:27.820935 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:27.820905    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40755b18-80e6-415f-8a27-4879ca6222b8" path="/var/lib/kubelet/pods/40755b18-80e6-415f-8a27-4879ca6222b8/volumes"
Oct 30 01:30:27.821752 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:27.821731    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce2530ca-6609-45c1-8438-17393a09e6c9" path="/var/lib/kubelet/pods/ce2530ca-6609-45c1-8438-17393a09e6c9/volumes"
Oct 30 01:30:28.424695 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:28.424653    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zz9kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 01:30:28.424906 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:28.424710    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn" podUID="0d79aa0b-3be0-4837-8182-e1e940a4d73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 01:30:28.431177 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:28.431144    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2h99j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 01:30:28.431317 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:28.431200    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j" podUID="01101e66-e714-4a99-af87-3e8e0e895867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 01:30:29.425961 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:29.425919    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zz9kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 01:30:29.426597 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:29.426549    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn" podUID="0d79aa0b-3be0-4837-8182-e1e940a4d73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 01:30:29.432342 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:29.432311    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2h99j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 01:30:29.432548 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:29.432523    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j" podUID="01101e66-e714-4a99-af87-3e8e0e895867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 01:30:30.424345 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:30.424305    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zz9kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 01:30:30.424595 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:30.424380    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn" podUID="0d79aa0b-3be0-4837-8182-e1e940a4d73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 01:30:31.336830 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:31.336028    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:31.337413 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:31.337128    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:31.338136 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:31.338113    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:30:31.721093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:31.721047    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:31.721620 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:31.721599    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:31.722305 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:30:31.722284    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:30:49.042923 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:30:49.042884    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:31:04.485668 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.485629    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv"]
Oct 30 01:31:04.486315 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486300    4864 cpu_manager.go:414] "containerMap: removing container" podUID="ce2530ca-6609-45c1-8438-17393a09e6c9" containerName="cleanup"
Oct 30 01:31:04.486366 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486316    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce2530ca-6609-45c1-8438-17393a09e6c9" containerName="cleanup"
Oct 30 01:31:04.486366 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486335    4864 cpu_manager.go:414] "containerMap: removing container" podUID="40755b18-80e6-415f-8a27-4879ca6222b8" containerName="cleanup"
Oct 30 01:31:04.486366 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486339    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40755b18-80e6-415f-8a27-4879ca6222b8" containerName="cleanup"
Oct 30 01:31:04.486366 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486362    4864 cpu_manager.go:414] "containerMap: removing container" podUID="56d653be-f408-4827-b79a-fba3572cc69d" containerName="cleanup"
Oct 30 01:31:04.486366 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486367    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d653be-f408-4827-b79a-fba3572cc69d" containerName="cleanup"
Oct 30 01:31:04.486522 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486373    4864 cpu_manager.go:414] "containerMap: removing container" podUID="abc4e617-7562-4f35-b084-aa303323866b" containerName="cleanup"
Oct 30 01:31:04.486522 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486378    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc4e617-7562-4f35-b084-aa303323866b" containerName="cleanup"
Oct 30 01:31:04.486522 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486384    4864 cpu_manager.go:414] "containerMap: removing container" podUID="40755b18-80e6-415f-8a27-4879ca6222b8" containerName="cleanup"
Oct 30 01:31:04.486522 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486389    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40755b18-80e6-415f-8a27-4879ca6222b8" containerName="cleanup"
Oct 30 01:31:04.486522 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486397    4864 cpu_manager.go:414] "containerMap: removing container" podUID="94b230f0-95fe-42ce-8fdf-8fea818268e3" containerName="cleanup"
Oct 30 01:31:04.486522 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486402    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94b230f0-95fe-42ce-8fdf-8fea818268e3" containerName="cleanup"
Oct 30 01:31:04.486522 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486409    4864 cpu_manager.go:414] "containerMap: removing container" podUID="94b230f0-95fe-42ce-8fdf-8fea818268e3" containerName="cleanup"
Oct 30 01:31:04.486522 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486413    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94b230f0-95fe-42ce-8fdf-8fea818268e3" containerName="cleanup"
Oct 30 01:31:04.486522 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486423    4864 cpu_manager.go:414] "containerMap: removing container" podUID="abc4e617-7562-4f35-b084-aa303323866b" containerName="cleanup"
Oct 30 01:31:04.486522 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486429    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc4e617-7562-4f35-b084-aa303323866b" containerName="cleanup"
Oct 30 01:31:04.486522 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486436    4864 cpu_manager.go:414] "containerMap: removing container" podUID="ce2530ca-6609-45c1-8438-17393a09e6c9" containerName="cleanup"
Oct 30 01:31:04.486522 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486440    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce2530ca-6609-45c1-8438-17393a09e6c9" containerName="cleanup"
Oct 30 01:31:04.486522 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486451    4864 cpu_manager.go:414] "containerMap: removing container" podUID="56d653be-f408-4827-b79a-fba3572cc69d" containerName="cleanup"
Oct 30 01:31:04.486522 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486455    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d653be-f408-4827-b79a-fba3572cc69d" containerName="cleanup"
Oct 30 01:31:04.486522 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486464    4864 cpu_manager.go:414] "containerMap: removing container" podUID="abc4e617-7562-4f35-b084-aa303323866b" containerName="cleanup"
Oct 30 01:31:04.486522 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486469    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc4e617-7562-4f35-b084-aa303323866b" containerName="cleanup"
Oct 30 01:31:04.486522 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486474    4864 cpu_manager.go:414] "containerMap: removing container" podUID="56d653be-f408-4827-b79a-fba3572cc69d" containerName="cleanup"
Oct 30 01:31:04.486522 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486478    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d653be-f408-4827-b79a-fba3572cc69d" containerName="cleanup"
Oct 30 01:31:04.487175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486572    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d653be-f408-4827-b79a-fba3572cc69d" containerName="cleanup"
Oct 30 01:31:04.487175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486581    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc4e617-7562-4f35-b084-aa303323866b" containerName="cleanup"
Oct 30 01:31:04.487175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486587    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="40755b18-80e6-415f-8a27-4879ca6222b8" containerName="cleanup"
Oct 30 01:31:04.487175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486601    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="94b230f0-95fe-42ce-8fdf-8fea818268e3" containerName="cleanup"
Oct 30 01:31:04.487175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486608    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="94b230f0-95fe-42ce-8fdf-8fea818268e3" containerName="cleanup"
Oct 30 01:31:04.487175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486613    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc4e617-7562-4f35-b084-aa303323866b" containerName="cleanup"
Oct 30 01:31:04.487175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486618    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce2530ca-6609-45c1-8438-17393a09e6c9" containerName="cleanup"
Oct 30 01:31:04.487175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486623    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="40755b18-80e6-415f-8a27-4879ca6222b8" containerName="cleanup"
Oct 30 01:31:04.487175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486628    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce2530ca-6609-45c1-8438-17393a09e6c9" containerName="cleanup"
Oct 30 01:31:04.487175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486633    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d653be-f408-4827-b79a-fba3572cc69d" containerName="cleanup"
Oct 30 01:31:04.487175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.486639    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d653be-f408-4827-b79a-fba3572cc69d" containerName="cleanup"
Oct 30 01:31:04.490243 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.490223    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv"
Oct 30 01:31:04.505885 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.505852    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-94ea1e\"/\"payload-processing-external-model-plugins-e2e-extmod-b-94ea1e\""
Oct 30 01:31:04.511624 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.511594    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv"]
Oct 30 01:31:04.516358 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.516336    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-94ea1e\"/\"openshift-service-ca.crt\""
Oct 30 01:31:04.516358 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.516338    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-94ea1e\"/\"routing-overlay\""
Oct 30 01:31:04.516749 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.516736    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-94ea1e\"/\"agc-cred-5c911d45fa08\""
Oct 30 01:31:04.516835 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.516752    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-94ea1e\"/\"agc-cred-fd829b22b320\""
Oct 30 01:31:04.516835 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.516820    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-94ea1e\"/\"payload-processing-external-model-e2e--33153ae7-dockercfg-6jxz7\""
Oct 30 01:31:04.585381 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.585342    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268c57fb-757f-4f7c-8194-ba036b41869b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv\" (UID: \"268c57fb-757f-4f7c-8194-ba036b41869b\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv"
Oct 30 01:31:04.585579 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.585467    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmlnl\" (UniqueName: \"kubernetes.io/projected/268c57fb-757f-4f7c-8194-ba036b41869b-kube-api-access-lmlnl\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv\" (UID: \"268c57fb-757f-4f7c-8194-ba036b41869b\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv"
Oct 30 01:31:04.585635 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.585594    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/268c57fb-757f-4f7c-8194-ba036b41869b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv\" (UID: \"268c57fb-757f-4f7c-8194-ba036b41869b\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv"
Oct 30 01:31:04.585750 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.585732    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/268c57fb-757f-4f7c-8194-ba036b41869b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv\" (UID: \"268c57fb-757f-4f7c-8194-ba036b41869b\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv"
Oct 30 01:31:04.602441 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.602405    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-94ea1e\"/\"kube-root-ca.crt\""
Oct 30 01:31:04.686773 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.686736    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268c57fb-757f-4f7c-8194-ba036b41869b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv\" (UID: \"268c57fb-757f-4f7c-8194-ba036b41869b\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv"
Oct 30 01:31:04.686954 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.686794    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmlnl\" (UniqueName: \"kubernetes.io/projected/268c57fb-757f-4f7c-8194-ba036b41869b-kube-api-access-lmlnl\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv\" (UID: \"268c57fb-757f-4f7c-8194-ba036b41869b\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv"
Oct 30 01:31:04.686954 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.686893    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/268c57fb-757f-4f7c-8194-ba036b41869b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv\" (UID: \"268c57fb-757f-4f7c-8194-ba036b41869b\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv"
Oct 30 01:31:04.687051 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.687030    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/268c57fb-757f-4f7c-8194-ba036b41869b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv\" (UID: \"268c57fb-757f-4f7c-8194-ba036b41869b\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv"
Oct 30 01:31:04.687564 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.687535    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268c57fb-757f-4f7c-8194-ba036b41869b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv\" (UID: \"268c57fb-757f-4f7c-8194-ba036b41869b\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv"
Oct 30 01:31:04.687674 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.687589    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/268c57fb-757f-4f7c-8194-ba036b41869b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv\" (UID: \"268c57fb-757f-4f7c-8194-ba036b41869b\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv"
Oct 30 01:31:04.689582 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.689555    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/268c57fb-757f-4f7c-8194-ba036b41869b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv\" (UID: \"268c57fb-757f-4f7c-8194-ba036b41869b\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv"
Oct 30 01:31:04.710574 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.710543    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmlnl\" (UniqueName: \"kubernetes.io/projected/268c57fb-757f-4f7c-8194-ba036b41869b-kube-api-access-lmlnl\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv\" (UID: \"268c57fb-757f-4f7c-8194-ba036b41869b\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv"
Oct 30 01:31:04.806313 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:04.806231    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv"
Oct 30 01:31:05.117533 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:05.117499    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv"]
Oct 30 01:31:05.118605 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:31:05.118580    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod268c57fb_757f_4f7c_8194_ba036b41869b.slice/crio-7a1598fc1b432171a1926eae552ecfe6027a7813b47608a44a4f368a0552119f WatchSource:0}: Error finding container 7a1598fc1b432171a1926eae552ecfe6027a7813b47608a44a4f368a0552119f: Status 404 returned error can't find the container with id 7a1598fc1b432171a1926eae552ecfe6027a7813b47608a44a4f368a0552119f
Oct 30 01:31:05.296978 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:05.296943    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv" event={"ID":"268c57fb-757f-4f7c-8194-ba036b41869b","Type":"ContainerStarted","Data":"aed914366117d24ed852ffb25344e2bbb357839e6bcef1adfadc96affb586640"}
Oct 30 01:31:05.297180 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:05.297005    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv"
Oct 30 01:31:05.297180 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:05.297025    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv" event={"ID":"268c57fb-757f-4f7c-8194-ba036b41869b","Type":"ContainerStarted","Data":"7a1598fc1b432171a1926eae552ecfe6027a7813b47608a44a4f368a0552119f"}
Oct 30 01:31:05.336752 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:05.336694    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv" podStartSLOduration=1.3366796 podStartE2EDuration="1.3366796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:31:05.335229161 +0000 UTC m=+2977.673331112" watchObservedRunningTime="2026-10-30 01:31:05.3366796 +0000 UTC m=+2977.674781509"
Oct 30 01:31:16.307178 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:16.307144    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv"
Oct 30 01:31:19.041609 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:31:19.041558    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:31:28.229489 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:28.229459    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:31:28.241008 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:28.240980    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:31:37.070137 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:37.070095    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"]
Oct 30 01:31:37.070743 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:37.070357    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk" podUID="48828a7f-0845-4165-9046-40358f733ba1" containerName="istio-proxy" containerID="cri-o://fe3c310093366327ebe676da12b7fd5e05e57d473480a7aa373e29b184925043" gracePeriod=30
Oct 30 01:31:42.418488 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.418457    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:31:42.518294 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.518247    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-credential-socket\") pod \"48828a7f-0845-4165-9046-40358f733ba1\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") "
Oct 30 01:31:42.518472 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.518327    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48828a7f-0845-4165-9046-40358f733ba1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48828a7f-0845-4165-9046-40358f733ba1-istiod-ca-cert\") pod \"48828a7f-0845-4165-9046-40358f733ba1\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") "
Oct 30 01:31:42.518472 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.518455    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48828a7f-0845-4165-9046-40358f733ba1-istio-token\" (UniqueName: \"kubernetes.io/projected/48828a7f-0845-4165-9046-40358f733ba1-istio-token\") pod \"48828a7f-0845-4165-9046-40358f733ba1\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") "
Oct 30 01:31:42.518560 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.518498    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-istio-envoy\") pod \"48828a7f-0845-4165-9046-40358f733ba1\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") "
Oct 30 01:31:42.518560 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.518532    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-istio-data\") pod \"48828a7f-0845-4165-9046-40358f733ba1\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") "
Oct 30 01:31:42.519122 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.518650    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48828a7f-0845-4165-9046-40358f733ba1-kube-api-access-slkmh\" (UniqueName: \"kubernetes.io/projected/48828a7f-0845-4165-9046-40358f733ba1-kube-api-access-slkmh\") pod \"48828a7f-0845-4165-9046-40358f733ba1\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") "
Oct 30 01:31:42.519122 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.518683    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-workload-socket\") pod \"48828a7f-0845-4165-9046-40358f733ba1\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") "
Oct 30 01:31:42.519122 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.519085    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-workload-socket" pod "48828a7f-0845-4165-9046-40358f733ba1" (UID: "48828a7f-0845-4165-9046-40358f733ba1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:42.519379 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.519336    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-credential-socket" pod "48828a7f-0845-4165-9046-40358f733ba1" (UID: "48828a7f-0845-4165-9046-40358f733ba1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:42.519698 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.519649    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48828a7f-0845-4165-9046-40358f733ba1-istiod-ca-cert" pod "48828a7f-0845-4165-9046-40358f733ba1" (UID: "48828a7f-0845-4165-9046-40358f733ba1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:42.520129 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.520104    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-istio-data" pod "48828a7f-0845-4165-9046-40358f733ba1" (UID: "48828a7f-0845-4165-9046-40358f733ba1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:42.520325 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.520306    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48828a7f-0845-4165-9046-40358f733ba1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48828a7f-0845-4165-9046-40358f733ba1-istio-podinfo\") pod \"48828a7f-0845-4165-9046-40358f733ba1\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") "
Oct 30 01:31:42.520857 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.520839    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-workload-certs\") pod \"48828a7f-0845-4165-9046-40358f733ba1\" (UID: \"48828a7f-0845-4165-9046-40358f733ba1\") "
Oct 30 01:31:42.521374 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.521190    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-workload-certs" pod "48828a7f-0845-4165-9046-40358f733ba1" (UID: "48828a7f-0845-4165-9046-40358f733ba1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:42.522130 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.522103    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-istio-envoy" pod "48828a7f-0845-4165-9046-40358f733ba1" (UID: "48828a7f-0845-4165-9046-40358f733ba1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:42.522485 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.522461    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:31:42.522589 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.522490    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:31:42.522589 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.522504    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48828a7f-0845-4165-9046-40358f733ba1-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:31:42.522589 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.522520    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:31:42.522589 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.522532    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:31:42.522589 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.522545    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48828a7f-0845-4165-9046-40358f733ba1-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:31:42.523805 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.523552    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48828a7f-0845-4165-9046-40358f733ba1-kube-api-access-slkmh" pod "48828a7f-0845-4165-9046-40358f733ba1" (UID: "48828a7f-0845-4165-9046-40358f733ba1"). InnerVolumeSpecName "kube-api-access-slkmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:42.523805 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.523763    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48828a7f-0845-4165-9046-40358f733ba1-istio-podinfo" pod "48828a7f-0845-4165-9046-40358f733ba1" (UID: "48828a7f-0845-4165-9046-40358f733ba1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:31:42.533585 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.533551    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48828a7f-0845-4165-9046-40358f733ba1-istio-token" pod "48828a7f-0845-4165-9046-40358f733ba1" (UID: "48828a7f-0845-4165-9046-40358f733ba1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:42.624713 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.624668    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48828a7f-0845-4165-9046-40358f733ba1-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:31:42.624713 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.624705    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48828a7f-0845-4165-9046-40358f733ba1-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:31:42.624916 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.624720    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slkmh\" (UniqueName: \"kubernetes.io/projected/48828a7f-0845-4165-9046-40358f733ba1-kube-api-access-slkmh\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:31:42.641632 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.641546    4864 generic.go:356] "Generic (PLEG): container finished" podID="48828a7f-0845-4165-9046-40358f733ba1" containerID="fe3c310093366327ebe676da12b7fd5e05e57d473480a7aa373e29b184925043" exitCode=0
Oct 30 01:31:42.641632 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.641592    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"
Oct 30 01:31:42.641632 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.641605    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk" event={"ID":"48828a7f-0845-4165-9046-40358f733ba1","Type":"ContainerDied","Data":"fe3c310093366327ebe676da12b7fd5e05e57d473480a7aa373e29b184925043"}
Oct 30 01:31:42.641885 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.641657    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk" event={"ID":"48828a7f-0845-4165-9046-40358f733ba1","Type":"ContainerDied","Data":"c34f400a76251a4441458d3c64154c1694c21024869bfa98358f68fe70d837ed"}
Oct 30 01:31:42.641885 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.641687    4864 scope.go:122] "RemoveContainer" containerID="fe3c310093366327ebe676da12b7fd5e05e57d473480a7aa373e29b184925043"
Oct 30 01:31:42.653789 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.653769    4864 scope.go:122] "RemoveContainer" containerID="fe3c310093366327ebe676da12b7fd5e05e57d473480a7aa373e29b184925043"
Oct 30 01:31:42.654156 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:31:42.654130    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe3c310093366327ebe676da12b7fd5e05e57d473480a7aa373e29b184925043\": container with ID starting with fe3c310093366327ebe676da12b7fd5e05e57d473480a7aa373e29b184925043 not found: ID does not exist" containerID="fe3c310093366327ebe676da12b7fd5e05e57d473480a7aa373e29b184925043"
Oct 30 01:31:42.654268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.654167    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe3c310093366327ebe676da12b7fd5e05e57d473480a7aa373e29b184925043"} err="failed to get container status \"fe3c310093366327ebe676da12b7fd5e05e57d473480a7aa373e29b184925043\": rpc error: code = NotFound desc = could not find container \"fe3c310093366327ebe676da12b7fd5e05e57d473480a7aa373e29b184925043\": container with ID starting with fe3c310093366327ebe676da12b7fd5e05e57d473480a7aa373e29b184925043 not found: ID does not exist"
Oct 30 01:31:42.696020 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.695981    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"]
Oct 30 01:31:42.708615 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:42.708579    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-d7zbk"]
Oct 30 01:31:43.795442 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:31:43.795402    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48828a7f-0845-4165-9046-40358f733ba1" path="/var/lib/kubelet/pods/48828a7f-0845-4165-9046-40358f733ba1/volumes"
Oct 30 01:31:49.055256 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:31:49.055214    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:32:07.013965 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.013929    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"]
Oct 30 01:32:07.014970 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.014942    4864 cpu_manager.go:414] "containerMap: removing container" podUID="48828a7f-0845-4165-9046-40358f733ba1" containerName="istio-proxy"
Oct 30 01:32:07.015103 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.014970    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48828a7f-0845-4165-9046-40358f733ba1" containerName="istio-proxy"
Oct 30 01:32:07.015103 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.014998    4864 cpu_manager.go:414] "containerMap: removing container" podUID="94b230f0-95fe-42ce-8fdf-8fea818268e3" containerName="cleanup"
Oct 30 01:32:07.015103 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.015006    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94b230f0-95fe-42ce-8fdf-8fea818268e3" containerName="cleanup"
Oct 30 01:32:07.015103 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.015026    4864 cpu_manager.go:414] "containerMap: removing container" podUID="ce2530ca-6609-45c1-8438-17393a09e6c9" containerName="cleanup"
Oct 30 01:32:07.015103 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.015035    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce2530ca-6609-45c1-8438-17393a09e6c9" containerName="cleanup"
Oct 30 01:32:07.015375 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.015107    4864 cpu_manager.go:414] "containerMap: removing container" podUID="40755b18-80e6-415f-8a27-4879ca6222b8" containerName="cleanup"
Oct 30 01:32:07.015375 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.015115    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40755b18-80e6-415f-8a27-4879ca6222b8" containerName="cleanup"
Oct 30 01:32:07.015375 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.015273    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="48828a7f-0845-4165-9046-40358f733ba1" containerName="istio-proxy"
Oct 30 01:32:07.015375 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.015286    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="40755b18-80e6-415f-8a27-4879ca6222b8" containerName="cleanup"
Oct 30 01:32:07.015375 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.015296    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="94b230f0-95fe-42ce-8fdf-8fea818268e3" containerName="cleanup"
Oct 30 01:32:07.015375 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.015304    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc4e617-7562-4f35-b084-aa303323866b" containerName="cleanup"
Oct 30 01:32:07.015375 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.015318    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce2530ca-6609-45c1-8438-17393a09e6c9" containerName="cleanup"
Oct 30 01:32:07.020124 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.020101    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.055578 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.055549    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"]
Oct 30 01:32:07.061650 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.061621    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-bd287f-openshift-default-dockercfg-cn7cx\""
Oct 30 01:32:07.220094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.217599    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-credential-socket\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.220094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.217832    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67hhf\" (UniqueName: \"kubernetes.io/projected/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-kube-api-access-67hhf\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.220094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.217878    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-data\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.220094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.217956    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istiod-ca-cert\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.220094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.217996    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-workload-socket\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.220094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.218035    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-token\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.220094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.218113    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-podinfo\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.220094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.218218    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-envoy\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.220094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.218260    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-workload-certs\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.320148 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.319452    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67hhf\" (UniqueName: \"kubernetes.io/projected/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-kube-api-access-67hhf\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.320148 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.319519    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-data\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.320148 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.319581    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istiod-ca-cert\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.320148 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.319617    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-workload-socket\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.320148 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.319659    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-token\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.320148 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.319713    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-podinfo\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.320148 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.319793    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-envoy\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.320148 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.319828    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-workload-certs\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.320148 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.319880    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-credential-socket\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.325093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.324510    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-workload-socket\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.325265 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.325142    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-data\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.325822 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.325794    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istiod-ca-cert\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.331092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.326294    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-credential-socket\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.331092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.326317    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-workload-certs\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.331583 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.331551    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-envoy\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.341698 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.341661    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-podinfo\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.378871 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.378798    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67hhf\" (UniqueName: \"kubernetes.io/projected/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-kube-api-access-67hhf\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.383093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.382333    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-token\") pod \"e2e-inf-a-bd287f-openshift-default-575f554785-gpglf\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") " pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.636136 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:07.636033    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:07.999999 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:32:07.999971    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c58c64b_ffc4_4ef6_b1fd_b60360eb37b5.slice/crio-156542469ecac54aa8f4d61d3466b39d7da4120f87095704f12b65f0c6cb8bf3 WatchSource:0}: Error finding container 156542469ecac54aa8f4d61d3466b39d7da4120f87095704f12b65f0c6cb8bf3: Status 404 returned error can't find the container with id 156542469ecac54aa8f4d61d3466b39d7da4120f87095704f12b65f0c6cb8bf3
Oct 30 01:32:08.000269 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:08.000247    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"]
Oct 30 01:32:08.002846 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:08.002817    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:32:08.002921 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:08.002886    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:32:08.002921 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:08.002917    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:32:08.970283 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:08.970244    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf" event={"ID":"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5","Type":"ContainerStarted","Data":"435baf638e3d19d13bf699c500285b51717ecba2bde80a7fefce4837a65ba236"}
Oct 30 01:32:08.970760 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:08.970313    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf" event={"ID":"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5","Type":"ContainerStarted","Data":"156542469ecac54aa8f4d61d3466b39d7da4120f87095704f12b65f0c6cb8bf3"}
Oct 30 01:32:09.025257 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:09.025208    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf" podStartSLOduration=3.025192279 podStartE2EDuration="3.025192279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:09.022315529 +0000 UTC m=+3041.360417438" watchObservedRunningTime="2026-10-30 01:32:09.025192279 +0000 UTC m=+3041.363294190"
Oct 30 01:32:09.636954 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:09.636922    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:09.642913 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:09.642887    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:09.977213 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:09.977183    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:09.978134 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:32:09.978114    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:32:19.042999 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:32:19.042958    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:32:49.042003 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:32:49.041961    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:33:19.044882 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:33:19.044799    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:33:36.726064 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:36.726028    4864 generic.go:356] "Generic (PLEG): container finished" podID="6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" containerID="435baf638e3d19d13bf699c500285b51717ecba2bde80a7fefce4837a65ba236" exitCode=0
Oct 30 01:33:36.726530 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:36.726102    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf" event={"ID":"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5","Type":"ContainerDied","Data":"435baf638e3d19d13bf699c500285b51717ecba2bde80a7fefce4837a65ba236"}
Oct 30 01:33:36.726595 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:36.726578    4864 scope.go:122] "RemoveContainer" containerID="435baf638e3d19d13bf699c500285b51717ecba2bde80a7fefce4837a65ba236"
Oct 30 01:33:36.728498 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:36.728464    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:33:36.728612 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:36.728553    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:33:36.728678 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:36.728606    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:33:37.748176 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:37.746013    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf" event={"ID":"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5","Type":"ContainerStarted","Data":"141e1d76ee6cf0f0238cf91bbe5fed92c12e621a7ea474ad428dc6f2cdbff24f"}
Oct 30 01:33:38.636807 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:38.636766    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:33:38.638139 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:38.638110    4864 patch_prober.go:29] interesting pod/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 01:33:38.638269 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:38.638172    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf" podUID="6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 01:33:39.636976 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:39.636938    4864 patch_prober.go:29] interesting pod/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 01:33:39.637438 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:39.637001    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf" podUID="6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 01:33:39.979096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:39.978438    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:33:40.637144 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:40.637105    4864 patch_prober.go:29] interesting pod/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 01:33:40.637636 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:40.637179    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf" podUID="6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 01:33:41.636669 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:41.636634    4864 patch_prober.go:29] interesting pod/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 01:33:41.636853 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:41.636690    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf" podUID="6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 01:33:42.636377 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:42.636343    4864 patch_prober.go:29] interesting pod/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 01:33:42.636774 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:42.636405    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf" podUID="6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 01:33:43.642190 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:43.642162    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:33:43.643342 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:43.643322    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:33:46.150977 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.150943    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"]
Oct 30 01:33:46.154632 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.154608    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.178105 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.178057    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"]
Oct 30 01:33:46.220467 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.220430    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ab0a2c-openshift-default-dockercfg-htqb8\""
Oct 30 01:33:46.302040 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.301999    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-credential-socket\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.306093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.302313    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55134196-9552-406d-931e-b95123dcbd94-istio-token\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.306444 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.306418    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-istio-data\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.321092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.306625    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jght\" (UniqueName: \"kubernetes.io/projected/55134196-9552-406d-931e-b95123dcbd94-kube-api-access-8jght\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.321705 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.321582    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55134196-9552-406d-931e-b95123dcbd94-istio-podinfo\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.322045 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.322021    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55134196-9552-406d-931e-b95123dcbd94-istiod-ca-cert\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.351097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.349210    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-istio-envoy\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.351097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.349291    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-workload-certs\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.351097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.349373    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-workload-socket\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.453610 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.451590    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55134196-9552-406d-931e-b95123dcbd94-istio-podinfo\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.453610 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.451651    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55134196-9552-406d-931e-b95123dcbd94-istiod-ca-cert\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.453610 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.451759    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-istio-envoy\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.453610 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.451804    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-workload-certs\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.453610 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.451868    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-workload-socket\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.453610 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.451976    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-credential-socket\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.453610 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.452027    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55134196-9552-406d-931e-b95123dcbd94-istio-token\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.453610 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.452117    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-istio-data\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.453610 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.452182    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jght\" (UniqueName: \"kubernetes.io/projected/55134196-9552-406d-931e-b95123dcbd94-kube-api-access-8jght\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.453610 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.452304    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-workload-certs\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.453610 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.452574    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-workload-socket\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.453610 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.452611    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-istio-data\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.453610 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.452933    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-credential-socket\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.454270 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.453657    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55134196-9552-406d-931e-b95123dcbd94-istiod-ca-cert\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.454616 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.454595    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-istio-envoy\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.455749 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.455730    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55134196-9552-406d-931e-b95123dcbd94-istio-podinfo\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.474057 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.474027    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55134196-9552-406d-931e-b95123dcbd94-istio-token\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.474450 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.474428    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jght\" (UniqueName: \"kubernetes.io/projected/55134196-9552-406d-931e-b95123dcbd94-kube-api-access-8jght\") pod \"e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") " pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:46.781565 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:46.781196    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:47.139948 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:47.139922    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"]
Oct 30 01:33:47.140863 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:33:47.140834    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55134196_9552_406d_931e_b95123dcbd94.slice/crio-e71afea8d3d7b88470f1bfaf2fe7b5f60e9ba9dc69d0a5b8e2ab3e199372d8cf WatchSource:0}: Error finding container e71afea8d3d7b88470f1bfaf2fe7b5f60e9ba9dc69d0a5b8e2ab3e199372d8cf: Status 404 returned error can't find the container with id e71afea8d3d7b88470f1bfaf2fe7b5f60e9ba9dc69d0a5b8e2ab3e199372d8cf
Oct 30 01:33:47.143851 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:47.143819    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:33:47.143931 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:47.143887    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:33:47.143931 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:47.143920    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:33:47.844226 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:47.844186    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl" event={"ID":"55134196-9552-406d-931e-b95123dcbd94","Type":"ContainerStarted","Data":"01ba5bdcf22c660eeb97c700ecae563ecf36e4ff93dd38f758d996d42b9a2957"}
Oct 30 01:33:47.844738 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:47.844239    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl" event={"ID":"55134196-9552-406d-931e-b95123dcbd94","Type":"ContainerStarted","Data":"e71afea8d3d7b88470f1bfaf2fe7b5f60e9ba9dc69d0a5b8e2ab3e199372d8cf"}
Oct 30 01:33:47.897991 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:47.897935    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl" podStartSLOduration=1.897920124 podStartE2EDuration="1.897920124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:47.895369295 +0000 UTC m=+3140.233471206" watchObservedRunningTime="2026-10-30 01:33:47.897920124 +0000 UTC m=+3140.236022108"
Oct 30 01:33:48.782982 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:48.782947    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:48.789757 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:48.789736    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:48.850906 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:48.850873    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:48.851887 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:48.851868    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:33:49.045132 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:33:49.045040    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:33:52.156368 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:52.156334    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl"]
Oct 30 01:33:52.160183 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:52.160160    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl"
Oct 30 01:33:52.179710 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:52.179685    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl"]
Oct 30 01:33:52.186101 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:52.186059    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ab0a2c\""
Oct 30 01:33:52.192990 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:52.192963    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/910c155f-e976-4ae1-bb59-7aa5cb000392-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl\" (UID: \"910c155f-e976-4ae1-bb59-7aa5cb000392\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl"
Oct 30 01:33:52.193154 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:52.193112    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqvz9\" (UniqueName: \"kubernetes.io/projected/910c155f-e976-4ae1-bb59-7aa5cb000392-kube-api-access-fqvz9\") pod \"payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl\" (UID: \"910c155f-e976-4ae1-bb59-7aa5cb000392\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl"
Oct 30 01:33:52.306959 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:52.298209    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/910c155f-e976-4ae1-bb59-7aa5cb000392-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl\" (UID: \"910c155f-e976-4ae1-bb59-7aa5cb000392\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl"
Oct 30 01:33:52.340224 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:52.334639    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqvz9\" (UniqueName: \"kubernetes.io/projected/910c155f-e976-4ae1-bb59-7aa5cb000392-kube-api-access-fqvz9\") pod \"payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl\" (UID: \"910c155f-e976-4ae1-bb59-7aa5cb000392\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl"
Oct 30 01:33:52.340224 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:52.308224    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/910c155f-e976-4ae1-bb59-7aa5cb000392-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl\" (UID: \"910c155f-e976-4ae1-bb59-7aa5cb000392\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl"
Oct 30 01:33:52.392492 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:52.373112    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqvz9\" (UniqueName: \"kubernetes.io/projected/910c155f-e976-4ae1-bb59-7aa5cb000392-kube-api-access-fqvz9\") pod \"payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl\" (UID: \"910c155f-e976-4ae1-bb59-7aa5cb000392\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl"
Oct 30 01:33:52.479096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:52.475315    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl"
Oct 30 01:33:52.803230 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:52.803138    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl"]
Oct 30 01:33:52.804323 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:33:52.804292    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod910c155f_e976_4ae1_bb59_7aa5cb000392.slice/crio-ab2cdb85cb13ab15b7d77ccc827e076359c28874ab199cd8caa5c34ea70644c9 WatchSource:0}: Error finding container ab2cdb85cb13ab15b7d77ccc827e076359c28874ab199cd8caa5c34ea70644c9: Status 404 returned error can't find the container with id ab2cdb85cb13ab15b7d77ccc827e076359c28874ab199cd8caa5c34ea70644c9
Oct 30 01:33:52.884762 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:52.884732    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl" event={"ID":"910c155f-e976-4ae1-bb59-7aa5cb000392","Type":"ContainerStarted","Data":"ab2cdb85cb13ab15b7d77ccc827e076359c28874ab199cd8caa5c34ea70644c9"}
Oct 30 01:33:53.893576 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:53.893527    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl" event={"ID":"910c155f-e976-4ae1-bb59-7aa5cb000392","Type":"ContainerStarted","Data":"e6f820e7458982caaef30425d57134ec5ae39c66ae210394725dd4abb7507e31"}
Oct 30 01:33:53.893978 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:53.893627    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl"
Oct 30 01:33:53.979199 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:53.979143    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl" podStartSLOduration=1.9791252030000002 podStartE2EDuration="1.979125203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:53.972796253 +0000 UTC m=+3146.310898174" watchObservedRunningTime="2026-10-30 01:33:53.979125203 +0000 UTC m=+3146.317227114"
Oct 30 01:33:55.577304 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:55.577270    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2"]
Oct 30 01:33:55.580854 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:55.580836    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2"
Oct 30 01:33:55.606590 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:55.606501    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ae1820\""
Oct 30 01:33:55.606590 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:55.606523    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"openshift-service-ca.crt\""
Oct 30 01:33:55.606590 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:55.606564    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"agc-cred-75b298c27b49\""
Oct 30 01:33:55.606885 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:55.606612    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"kube-root-ca.crt\""
Oct 30 01:33:55.606942 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:55.606929    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"routing-overlay\""
Oct 30 01:33:55.607059 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:55.607044    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"agc-cred-61bd2e792be2\""
Oct 30 01:33:55.608268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:55.608247    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"payload-processing-external-model-e2e--53969a13-dockercfg-24pcd\""
Oct 30 01:33:55.610832 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:55.610764    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2"]
Oct 30 01:33:55.700623 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:55.700585    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c7f591-fa59-4890-a139-240501e03109-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2\" (UID: \"54c7f591-fa59-4890-a139-240501e03109\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2"
Oct 30 01:33:55.700839 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:55.700663    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54c7f591-fa59-4890-a139-240501e03109-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2\" (UID: \"54c7f591-fa59-4890-a139-240501e03109\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2"
Oct 30 01:33:55.700914 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:55.700854    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54c7f591-fa59-4890-a139-240501e03109-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2\" (UID: \"54c7f591-fa59-4890-a139-240501e03109\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2"
Oct 30 01:33:55.701030 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:55.701015    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssg6p\" (UniqueName: \"kubernetes.io/projected/54c7f591-fa59-4890-a139-240501e03109-kube-api-access-ssg6p\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2\" (UID: \"54c7f591-fa59-4890-a139-240501e03109\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2"
Oct 30 01:33:55.802844 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:55.802807    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54c7f591-fa59-4890-a139-240501e03109-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2\" (UID: \"54c7f591-fa59-4890-a139-240501e03109\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2"
Oct 30 01:33:55.803000 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:55.802968    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54c7f591-fa59-4890-a139-240501e03109-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2\" (UID: \"54c7f591-fa59-4890-a139-240501e03109\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2"
Oct 30 01:33:55.803145 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:55.803127    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssg6p\" (UniqueName: \"kubernetes.io/projected/54c7f591-fa59-4890-a139-240501e03109-kube-api-access-ssg6p\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2\" (UID: \"54c7f591-fa59-4890-a139-240501e03109\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2"
Oct 30 01:33:55.803241 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:55.803224    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c7f591-fa59-4890-a139-240501e03109-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2\" (UID: \"54c7f591-fa59-4890-a139-240501e03109\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2"
Oct 30 01:33:55.803584 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:55.803561    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54c7f591-fa59-4890-a139-240501e03109-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2\" (UID: \"54c7f591-fa59-4890-a139-240501e03109\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2"
Oct 30 01:33:55.803950 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:55.803926    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c7f591-fa59-4890-a139-240501e03109-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2\" (UID: \"54c7f591-fa59-4890-a139-240501e03109\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2"
Oct 30 01:33:55.805685 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:55.805662    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54c7f591-fa59-4890-a139-240501e03109-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2\" (UID: \"54c7f591-fa59-4890-a139-240501e03109\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2"
Oct 30 01:33:55.827017 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:55.826987    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssg6p\" (UniqueName: \"kubernetes.io/projected/54c7f591-fa59-4890-a139-240501e03109-kube-api-access-ssg6p\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2\" (UID: \"54c7f591-fa59-4890-a139-240501e03109\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2"
Oct 30 01:33:55.894773 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:55.894686    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2"
Oct 30 01:33:56.207185 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:56.207154    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2"]
Oct 30 01:33:56.208662 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:33:56.208638    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c7f591_fa59_4890_a139_240501e03109.slice/crio-e0eb169a10b1afc01fa4cdc31f16d5f957fa50ea1a55d78d8632a63ba09baa6d WatchSource:0}: Error finding container e0eb169a10b1afc01fa4cdc31f16d5f957fa50ea1a55d78d8632a63ba09baa6d: Status 404 returned error can't find the container with id e0eb169a10b1afc01fa4cdc31f16d5f957fa50ea1a55d78d8632a63ba09baa6d
Oct 30 01:33:56.921939 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:56.921898    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2" event={"ID":"54c7f591-fa59-4890-a139-240501e03109","Type":"ContainerStarted","Data":"29703e2391963a9c9ee6de0748f99cb516870a2dbaba05dae18c8e68dd0bec77"}
Oct 30 01:33:56.922422 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:56.921966    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2" event={"ID":"54c7f591-fa59-4890-a139-240501e03109","Type":"ContainerStarted","Data":"e0eb169a10b1afc01fa4cdc31f16d5f957fa50ea1a55d78d8632a63ba09baa6d"}
Oct 30 01:33:56.922422 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:56.922004    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2"
Oct 30 01:33:56.965010 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:56.964958    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2" podStartSLOduration=1.9649435020000001 podStartE2EDuration="1.964943502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:56.960945477 +0000 UTC m=+3149.299047386" watchObservedRunningTime="2026-10-30 01:33:56.964943502 +0000 UTC m=+3149.303045414"
Oct 30 01:33:57.020421 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.020390    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22"]
Oct 30 01:33:57.024141 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.024112    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22"
Oct 30 01:33:57.040464 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.040375    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a47b9a\"/\"routing-overlay\""
Oct 30 01:33:57.040464 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.040380    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a47b9a\"/\"agc-cred-5ffddfb0a62f\""
Oct 30 01:33:57.040464 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.040384    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a47b9a\"/\"agc-cred-8f41cd4e896a\""
Oct 30 01:33:57.040464 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.040465    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a47b9a\"/\"kube-root-ca.crt\""
Oct 30 01:33:57.040729 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.040505    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a47b9a\"/\"openshift-service-ca.crt\""
Oct 30 01:33:57.040729 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.040656    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a47b9a\"/\"payload-processing-external-model-e2e--beface97-dockercfg-9j667\""
Oct 30 01:33:57.040880 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.040857    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a47b9a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a47b9a\""
Oct 30 01:33:57.042304 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.042284    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22"]
Oct 30 01:33:57.124781 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.124738    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80dbf21d-7985-47dc-8b73-2a13e0a0399c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22\" (UID: \"80dbf21d-7985-47dc-8b73-2a13e0a0399c\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22"
Oct 30 01:33:57.125014 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.124981    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84wzq\" (UniqueName: \"kubernetes.io/projected/80dbf21d-7985-47dc-8b73-2a13e0a0399c-kube-api-access-84wzq\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22\" (UID: \"80dbf21d-7985-47dc-8b73-2a13e0a0399c\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22"
Oct 30 01:33:57.125058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.125034    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80dbf21d-7985-47dc-8b73-2a13e0a0399c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22\" (UID: \"80dbf21d-7985-47dc-8b73-2a13e0a0399c\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22"
Oct 30 01:33:57.125216 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.125200    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80dbf21d-7985-47dc-8b73-2a13e0a0399c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22\" (UID: \"80dbf21d-7985-47dc-8b73-2a13e0a0399c\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22"
Oct 30 01:33:57.227140 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.227102    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80dbf21d-7985-47dc-8b73-2a13e0a0399c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22\" (UID: \"80dbf21d-7985-47dc-8b73-2a13e0a0399c\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22"
Oct 30 01:33:57.227318 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.227189    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80dbf21d-7985-47dc-8b73-2a13e0a0399c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22\" (UID: \"80dbf21d-7985-47dc-8b73-2a13e0a0399c\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22"
Oct 30 01:33:57.227362 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.227343    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84wzq\" (UniqueName: \"kubernetes.io/projected/80dbf21d-7985-47dc-8b73-2a13e0a0399c-kube-api-access-84wzq\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22\" (UID: \"80dbf21d-7985-47dc-8b73-2a13e0a0399c\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22"
Oct 30 01:33:57.227405 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.227367    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80dbf21d-7985-47dc-8b73-2a13e0a0399c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22\" (UID: \"80dbf21d-7985-47dc-8b73-2a13e0a0399c\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22"
Oct 30 01:33:57.227910 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.227884    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80dbf21d-7985-47dc-8b73-2a13e0a0399c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22\" (UID: \"80dbf21d-7985-47dc-8b73-2a13e0a0399c\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22"
Oct 30 01:33:57.228007 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.227959    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80dbf21d-7985-47dc-8b73-2a13e0a0399c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22\" (UID: \"80dbf21d-7985-47dc-8b73-2a13e0a0399c\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22"
Oct 30 01:33:57.229962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.229941    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80dbf21d-7985-47dc-8b73-2a13e0a0399c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22\" (UID: \"80dbf21d-7985-47dc-8b73-2a13e0a0399c\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22"
Oct 30 01:33:57.249785 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.249759    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84wzq\" (UniqueName: \"kubernetes.io/projected/80dbf21d-7985-47dc-8b73-2a13e0a0399c-kube-api-access-84wzq\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22\" (UID: \"80dbf21d-7985-47dc-8b73-2a13e0a0399c\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22"
Oct 30 01:33:57.341775 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.341742    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22"
Oct 30 01:33:57.664209 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.664182    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22"]
Oct 30 01:33:57.932584 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.932481    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22" event={"ID":"80dbf21d-7985-47dc-8b73-2a13e0a0399c","Type":"ContainerStarted","Data":"4aece0cac57f0d7c23fed9758aec0eb9195e523c4290d13ad62211f023118011"}
Oct 30 01:33:57.932584 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.932546    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22" event={"ID":"80dbf21d-7985-47dc-8b73-2a13e0a0399c","Type":"ContainerStarted","Data":"adaedd5a2eb0cc3da62639ff811cfca0781d58f0da68e9185e9af65b5319461b"}
Oct 30 01:33:57.932584 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.932575    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22"
Oct 30 01:33:57.973500 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:33:57.973440    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22" podStartSLOduration=1.9734199270000001 podStartE2EDuration="1.973419927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:57.97076107 +0000 UTC m=+3150.308862992" watchObservedRunningTime="2026-10-30 01:33:57.973419927 +0000 UTC m=+3150.311521839"
Oct 30 01:34:04.907480 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:04.907447    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl"
Oct 30 01:34:07.962094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:07.960383    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2"
Oct 30 01:34:08.943289 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:08.943262    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22"
Oct 30 01:34:19.044565 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:34:19.044518    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:34:19.209138 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:19.209104    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv"]
Oct 30 01:34:24.209671 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:24.209621    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv" podUID="268c57fb-757f-4f7c-8194-ba036b41869b" containerName="payload-processing" containerID="cri-o://aed914366117d24ed852ffb25344e2bbb357839e6bcef1adfadc96affb586640" gracePeriod=25
Oct 30 01:34:24.490557 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:24.490529    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv"
Oct 30 01:34:24.559506 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:24.559472    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/268c57fb-757f-4f7c-8194-ba036b41869b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/268c57fb-757f-4f7c-8194-ba036b41869b-provider-credentials\") pod \"268c57fb-757f-4f7c-8194-ba036b41869b\" (UID: \"268c57fb-757f-4f7c-8194-ba036b41869b\") "
Oct 30 01:34:24.559713 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:24.559571    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/268c57fb-757f-4f7c-8194-ba036b41869b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268c57fb-757f-4f7c-8194-ba036b41869b-plugins-config-volume\") pod \"268c57fb-757f-4f7c-8194-ba036b41869b\" (UID: \"268c57fb-757f-4f7c-8194-ba036b41869b\") "
Oct 30 01:34:24.559713 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:24.559637    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/268c57fb-757f-4f7c-8194-ba036b41869b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/268c57fb-757f-4f7c-8194-ba036b41869b-routing-overlay-volume\") pod \"268c57fb-757f-4f7c-8194-ba036b41869b\" (UID: \"268c57fb-757f-4f7c-8194-ba036b41869b\") "
Oct 30 01:34:24.559713 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:24.559664    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/268c57fb-757f-4f7c-8194-ba036b41869b-kube-api-access-lmlnl\" (UniqueName: \"kubernetes.io/projected/268c57fb-757f-4f7c-8194-ba036b41869b-kube-api-access-lmlnl\") pod \"268c57fb-757f-4f7c-8194-ba036b41869b\" (UID: \"268c57fb-757f-4f7c-8194-ba036b41869b\") "
Oct 30 01:34:24.560453 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:24.560010    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/268c57fb-757f-4f7c-8194-ba036b41869b-routing-overlay-volume" pod "268c57fb-757f-4f7c-8194-ba036b41869b" (UID: "268c57fb-757f-4f7c-8194-ba036b41869b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:34:24.560453 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:24.560045    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/268c57fb-757f-4f7c-8194-ba036b41869b-plugins-config-volume" pod "268c57fb-757f-4f7c-8194-ba036b41869b" (UID: "268c57fb-757f-4f7c-8194-ba036b41869b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:34:24.560453 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:24.560402    4864 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268c57fb-757f-4f7c-8194-ba036b41869b-plugins-config-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:34:24.560453 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:24.560427    4864 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/268c57fb-757f-4f7c-8194-ba036b41869b-routing-overlay-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:34:24.562007 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:24.561976    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/268c57fb-757f-4f7c-8194-ba036b41869b-provider-credentials" pod "268c57fb-757f-4f7c-8194-ba036b41869b" (UID: "268c57fb-757f-4f7c-8194-ba036b41869b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:34:24.562256 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:24.562223    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/268c57fb-757f-4f7c-8194-ba036b41869b-kube-api-access-lmlnl" pod "268c57fb-757f-4f7c-8194-ba036b41869b" (UID: "268c57fb-757f-4f7c-8194-ba036b41869b"). InnerVolumeSpecName "kube-api-access-lmlnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:34:24.661972 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:24.661935    4864 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/268c57fb-757f-4f7c-8194-ba036b41869b-provider-credentials\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:34:24.661972 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:24.661967    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmlnl\" (UniqueName: \"kubernetes.io/projected/268c57fb-757f-4f7c-8194-ba036b41869b-kube-api-access-lmlnl\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:34:25.243670 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:25.243633    4864 generic.go:356] "Generic (PLEG): container finished" podID="268c57fb-757f-4f7c-8194-ba036b41869b" containerID="aed914366117d24ed852ffb25344e2bbb357839e6bcef1adfadc96affb586640" exitCode=0
Oct 30 01:34:25.244148 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:25.243681    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv"
Oct 30 01:34:25.244148 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:25.243723    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv" event={"ID":"268c57fb-757f-4f7c-8194-ba036b41869b","Type":"ContainerDied","Data":"aed914366117d24ed852ffb25344e2bbb357839e6bcef1adfadc96affb586640"}
Oct 30 01:34:25.244148 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:25.243780    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv" event={"ID":"268c57fb-757f-4f7c-8194-ba036b41869b","Type":"ContainerDied","Data":"7a1598fc1b432171a1926eae552ecfe6027a7813b47608a44a4f368a0552119f"}
Oct 30 01:34:25.244148 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:25.243808    4864 scope.go:122] "RemoveContainer" containerID="aed914366117d24ed852ffb25344e2bbb357839e6bcef1adfadc96affb586640"
Oct 30 01:34:25.259795 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:25.259767    4864 scope.go:122] "RemoveContainer" containerID="aed914366117d24ed852ffb25344e2bbb357839e6bcef1adfadc96affb586640"
Oct 30 01:34:25.263065 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:34:25.263038    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aed914366117d24ed852ffb25344e2bbb357839e6bcef1adfadc96affb586640\": container with ID starting with aed914366117d24ed852ffb25344e2bbb357839e6bcef1adfadc96affb586640 not found: ID does not exist" containerID="aed914366117d24ed852ffb25344e2bbb357839e6bcef1adfadc96affb586640"
Oct 30 01:34:25.263213 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:25.263108    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aed914366117d24ed852ffb25344e2bbb357839e6bcef1adfadc96affb586640"} err="failed to get container status \"aed914366117d24ed852ffb25344e2bbb357839e6bcef1adfadc96affb586640\": rpc error: code = NotFound desc = could not find container \"aed914366117d24ed852ffb25344e2bbb357839e6bcef1adfadc96affb586640\": container with ID starting with aed914366117d24ed852ffb25344e2bbb357839e6bcef1adfadc96affb586640 not found: ID does not exist"
Oct 30 01:34:25.348316 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:25.348277    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv"]
Oct 30 01:34:25.401160 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:25.401121    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5688x88nv"]
Oct 30 01:34:25.796054 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:25.796017    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="268c57fb-757f-4f7c-8194-ba036b41869b" path="/var/lib/kubelet/pods/268c57fb-757f-4f7c-8194-ba036b41869b/volumes"
Oct 30 01:34:26.748127 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:26.748085    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"]
Oct 30 01:34:26.748905 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:26.748881    4864 cpu_manager.go:414] "containerMap: removing container" podUID="268c57fb-757f-4f7c-8194-ba036b41869b" containerName="payload-processing"
Oct 30 01:34:26.748905 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:26.748904    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="268c57fb-757f-4f7c-8194-ba036b41869b" containerName="payload-processing"
Oct 30 01:34:26.749142 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:26.749107    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="268c57fb-757f-4f7c-8194-ba036b41869b" containerName="payload-processing"
Oct 30 01:34:26.754470 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:26.754449    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:26.775823 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:26.775791    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"]
Oct 30 01:34:26.830736 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:26.830694    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b29d4a-openshift-default-dockercfg-lpccr\""
Oct 30 01:34:26.912117 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:26.912025    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-workload-socket\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:26.912117 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:26.912113    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-credential-socket\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:26.912383 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:26.912174    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09577058-17aa-411b-bf71-092730f470f2-istiod-ca-cert\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:26.912383 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:26.912212    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nh58\" (UniqueName: \"kubernetes.io/projected/09577058-17aa-411b-bf71-092730f470f2-kube-api-access-6nh58\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:26.912383 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:26.912253    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09577058-17aa-411b-bf71-092730f470f2-istio-podinfo\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:26.912383 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:26.912297    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-workload-certs\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:26.912621 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:26.912393    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-istio-envoy\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:26.912621 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:26.912472    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-istio-data\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:26.912769 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:26.912745    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09577058-17aa-411b-bf71-092730f470f2-istio-token\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:27.025175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:27.015967    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-istio-data\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:27.025175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:27.016211    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09577058-17aa-411b-bf71-092730f470f2-istio-token\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:27.025175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:27.016244    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-workload-socket\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:27.025175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:27.016275    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-credential-socket\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:27.025175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:27.016321    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09577058-17aa-411b-bf71-092730f470f2-istiod-ca-cert\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:27.025175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:27.016350    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nh58\" (UniqueName: \"kubernetes.io/projected/09577058-17aa-411b-bf71-092730f470f2-kube-api-access-6nh58\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:27.025175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:27.016398    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09577058-17aa-411b-bf71-092730f470f2-istio-podinfo\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:27.025175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:27.016439    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-workload-certs\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:27.025175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:27.016516    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-istio-envoy\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:27.025175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:27.017375    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-credential-socket\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:27.025175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:27.017590    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-istio-data\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:27.025175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:27.018098    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-workload-socket\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:27.025175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:27.018877    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09577058-17aa-411b-bf71-092730f470f2-istiod-ca-cert\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:27.042158 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:27.038790    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-workload-certs\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:27.056112 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:27.044877    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09577058-17aa-411b-bf71-092730f470f2-istio-podinfo\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:27.068134 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:27.068050    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09577058-17aa-411b-bf71-092730f470f2-istio-token\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:27.069475 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:27.069452    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nh58\" (UniqueName: \"kubernetes.io/projected/09577058-17aa-411b-bf71-092730f470f2-kube-api-access-6nh58\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:27.076373 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:27.076338    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-istio-envoy\") pod \"e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") " pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:27.076888 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:27.076867    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:27.446700 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:27.446667    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"]
Oct 30 01:34:27.447904 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:34:27.447868    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09577058_17aa_411b_bf71_092730f470f2.slice/crio-ea8359d16dcb192b4ce08c9587cc185880a20296b33cbfa0ecfb88b33c020162 WatchSource:0}: Error finding container ea8359d16dcb192b4ce08c9587cc185880a20296b33cbfa0ecfb88b33c020162: Status 404 returned error can't find the container with id ea8359d16dcb192b4ce08c9587cc185880a20296b33cbfa0ecfb88b33c020162
Oct 30 01:34:27.450895 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:27.450863    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:34:27.450950 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:27.450928    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:34:27.450987 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:27.450957    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:34:28.268103 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:28.268031    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb" event={"ID":"09577058-17aa-411b-bf71-092730f470f2","Type":"ContainerStarted","Data":"56af3c0c24f8558222890a96e15ced297db12fe990addeaf2469802fc725f416"}
Oct 30 01:34:28.268526 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:28.268114    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb" event={"ID":"09577058-17aa-411b-bf71-092730f470f2","Type":"ContainerStarted","Data":"ea8359d16dcb192b4ce08c9587cc185880a20296b33cbfa0ecfb88b33c020162"}
Oct 30 01:34:28.314934 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:28.314866    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb" podStartSLOduration=2.314848541 podStartE2EDuration="2.314848541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:28.309352586 +0000 UTC m=+3180.647454489" watchObservedRunningTime="2026-10-30 01:34:28.314848541 +0000 UTC m=+3180.652950531"
Oct 30 01:34:29.077251 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:29.077213    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:29.083292 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:29.083266    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:29.276870 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:29.276833    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:29.278174 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:34:29.278147    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:34:49.080723 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:34:49.080636    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:35:10.030035 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:10.030001    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6702c2fe-tl2ld"]
Oct 30 01:35:10.039766 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:10.039732    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6702c2fe-tl2ld"]
Oct 30 01:35:11.795025 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:11.794988    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea5291f2-fd1f-433f-8cd1-986568a5be4e" path="/var/lib/kubelet/pods/ea5291f2-fd1f-433f-8cd1-986568a5be4e/volumes"
Oct 30 01:35:19.044049 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:35:19.044008    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:35:30.072255 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:30.072220    4864 generic.go:356] "Generic (PLEG): container finished" podID="55134196-9552-406d-931e-b95123dcbd94" containerID="01ba5bdcf22c660eeb97c700ecae563ecf36e4ff93dd38f758d996d42b9a2957" exitCode=0
Oct 30 01:35:30.072693 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:30.072292    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl" event={"ID":"55134196-9552-406d-931e-b95123dcbd94","Type":"ContainerDied","Data":"01ba5bdcf22c660eeb97c700ecae563ecf36e4ff93dd38f758d996d42b9a2957"}
Oct 30 01:35:30.072752 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:30.072736    4864 scope.go:122] "RemoveContainer" containerID="01ba5bdcf22c660eeb97c700ecae563ecf36e4ff93dd38f758d996d42b9a2957"
Oct 30 01:35:30.074769 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:30.074742    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:35:30.074836 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:30.074809    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:35:30.074876 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:30.074839    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:35:31.090129 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:31.090058    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl" event={"ID":"55134196-9552-406d-931e-b95123dcbd94","Type":"ContainerStarted","Data":"cc58e936ae5a72fb5121595962a6ab14e30e73449a9f4ead9ef808c8601f5cad"}
Oct 30 01:35:31.783016 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:31.782955    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:35:31.784506 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:31.784472    4864 patch_prober.go:29] interesting pod/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 01:35:31.784639 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:31.784544    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl" podUID="55134196-9552-406d-931e-b95123dcbd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 01:35:32.783604 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:32.783558    4864 patch_prober.go:29] interesting pod/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 01:35:32.784008 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:32.783635    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl" podUID="55134196-9552-406d-931e-b95123dcbd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 01:35:33.783380 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:33.783338    4864 patch_prober.go:29] interesting pod/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 01:35:33.783574 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:33.783410    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl" podUID="55134196-9552-406d-931e-b95123dcbd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 01:35:33.851181 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:33.851150    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:35:34.783286 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:34.783239    4864 patch_prober.go:29] interesting pod/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 01:35:34.783480 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:34.783309    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl" podUID="55134196-9552-406d-931e-b95123dcbd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 01:35:35.783163 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:35.783126    4864 patch_prober.go:29] interesting pod/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 01:35:35.783607 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:35.783189    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl" podUID="55134196-9552-406d-931e-b95123dcbd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 01:35:36.788473 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:36.788441    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:35:36.789894 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:36.789869    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:35:44.654890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.654856    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"]
Oct 30 01:35:44.658956 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.658932    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:35:44.684761 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.684717    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-415c6b\"/\"default-dockercfg-k6jgn\""
Oct 30 01:35:44.684761 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.684746    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-415c6b\"/\"kube-root-ca.crt\""
Oct 30 01:35:44.685011 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.684730    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-415c6b\"/\"openshift-service-ca.crt\""
Oct 30 01:35:44.685267 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.685246    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-415c6b\"/\"test-model-415c6b-kserve-self-signed-certs\""
Oct 30 01:35:44.691930 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.691869    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"]
Oct 30 01:35:44.707029 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.706996    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-home\") pod \"test-model-415c6b-kserve-85cd844477-lwrv9\" (UID: \"916074d0-6586-417a-baf6-df529fdd58b8\") " pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:35:44.707216 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.707142    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-tmp-dir\") pod \"test-model-415c6b-kserve-85cd844477-lwrv9\" (UID: \"916074d0-6586-417a-baf6-df529fdd58b8\") " pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:35:44.707216 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.707203    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-dshm\") pod \"test-model-415c6b-kserve-85cd844477-lwrv9\" (UID: \"916074d0-6586-417a-baf6-df529fdd58b8\") " pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:35:44.707322 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.707226    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-model-cache\") pod \"test-model-415c6b-kserve-85cd844477-lwrv9\" (UID: \"916074d0-6586-417a-baf6-df529fdd58b8\") " pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:35:44.707391 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.707372    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-547f6\" (UniqueName: \"kubernetes.io/projected/916074d0-6586-417a-baf6-df529fdd58b8-kube-api-access-547f6\") pod \"test-model-415c6b-kserve-85cd844477-lwrv9\" (UID: \"916074d0-6586-417a-baf6-df529fdd58b8\") " pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:35:44.707634 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.707612    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/916074d0-6586-417a-baf6-df529fdd58b8-tls-certs\") pod \"test-model-415c6b-kserve-85cd844477-lwrv9\" (UID: \"916074d0-6586-417a-baf6-df529fdd58b8\") " pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:35:44.808694 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.808655    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-tmp-dir\") pod \"test-model-415c6b-kserve-85cd844477-lwrv9\" (UID: \"916074d0-6586-417a-baf6-df529fdd58b8\") " pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:35:44.808884 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.808720    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-dshm\") pod \"test-model-415c6b-kserve-85cd844477-lwrv9\" (UID: \"916074d0-6586-417a-baf6-df529fdd58b8\") " pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:35:44.808884 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.808742    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-model-cache\") pod \"test-model-415c6b-kserve-85cd844477-lwrv9\" (UID: \"916074d0-6586-417a-baf6-df529fdd58b8\") " pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:35:44.808884 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.808816    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-547f6\" (UniqueName: \"kubernetes.io/projected/916074d0-6586-417a-baf6-df529fdd58b8-kube-api-access-547f6\") pod \"test-model-415c6b-kserve-85cd844477-lwrv9\" (UID: \"916074d0-6586-417a-baf6-df529fdd58b8\") " pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:35:44.808884 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.808859    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/916074d0-6586-417a-baf6-df529fdd58b8-tls-certs\") pod \"test-model-415c6b-kserve-85cd844477-lwrv9\" (UID: \"916074d0-6586-417a-baf6-df529fdd58b8\") " pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:35:44.809216 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.809019    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-home\") pod \"test-model-415c6b-kserve-85cd844477-lwrv9\" (UID: \"916074d0-6586-417a-baf6-df529fdd58b8\") " pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:35:44.809216 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.809119    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-tmp-dir\") pod \"test-model-415c6b-kserve-85cd844477-lwrv9\" (UID: \"916074d0-6586-417a-baf6-df529fdd58b8\") " pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:35:44.809318 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.809234    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-model-cache\") pod \"test-model-415c6b-kserve-85cd844477-lwrv9\" (UID: \"916074d0-6586-417a-baf6-df529fdd58b8\") " pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:35:44.809414 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.809395    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-home\") pod \"test-model-415c6b-kserve-85cd844477-lwrv9\" (UID: \"916074d0-6586-417a-baf6-df529fdd58b8\") " pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:35:44.810963 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.810936    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-dshm\") pod \"test-model-415c6b-kserve-85cd844477-lwrv9\" (UID: \"916074d0-6586-417a-baf6-df529fdd58b8\") " pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:35:44.811505 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.811482    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/916074d0-6586-417a-baf6-df529fdd58b8-tls-certs\") pod \"test-model-415c6b-kserve-85cd844477-lwrv9\" (UID: \"916074d0-6586-417a-baf6-df529fdd58b8\") " pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:35:44.830223 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.830194    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-547f6\" (UniqueName: \"kubernetes.io/projected/916074d0-6586-417a-baf6-df529fdd58b8-kube-api-access-547f6\") pod \"test-model-415c6b-kserve-85cd844477-lwrv9\" (UID: \"916074d0-6586-417a-baf6-df529fdd58b8\") " pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:35:44.973478 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:44.973437    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:35:45.349873 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:45.349792    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"]
Oct 30 01:35:45.355643 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:35:45.355610    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod916074d0_6586_417a_baf6_df529fdd58b8.slice/crio-b57d88f2593d19490d0759cff10f72eaa388765f9b0366b77d7ff4e0639e5365 WatchSource:0}: Error finding container b57d88f2593d19490d0759cff10f72eaa388765f9b0366b77d7ff4e0639e5365: Status 404 returned error can't find the container with id b57d88f2593d19490d0759cff10f72eaa388765f9b0366b77d7ff4e0639e5365
Oct 30 01:35:45.797695 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:45.797665    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj"]
Oct 30 01:35:45.801119 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:45.801097    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj"
Oct 30 01:35:45.819736 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:45.819704    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj"]
Oct 30 01:35:45.878202 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:45.878158    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj\" (UID: \"adef1f71-a5b9-4715-b3d8-f1c80bda8d8f\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj"
Oct 30 01:35:45.878395 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:45.878362    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj\" (UID: \"adef1f71-a5b9-4715-b3d8-f1c80bda8d8f\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj"
Oct 30 01:35:45.878679 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:45.878654    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj\" (UID: \"adef1f71-a5b9-4715-b3d8-f1c80bda8d8f\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj"
Oct 30 01:35:45.878780 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:45.878736    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmbmw\" (UniqueName: \"kubernetes.io/projected/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-kube-api-access-bmbmw\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj\" (UID: \"adef1f71-a5b9-4715-b3d8-f1c80bda8d8f\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj"
Oct 30 01:35:45.979931 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:45.979899    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj\" (UID: \"adef1f71-a5b9-4715-b3d8-f1c80bda8d8f\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj"
Oct 30 01:35:45.980171 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:45.979947    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmbmw\" (UniqueName: \"kubernetes.io/projected/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-kube-api-access-bmbmw\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj\" (UID: \"adef1f71-a5b9-4715-b3d8-f1c80bda8d8f\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj"
Oct 30 01:35:45.980171 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:45.980103    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj\" (UID: \"adef1f71-a5b9-4715-b3d8-f1c80bda8d8f\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj"
Oct 30 01:35:45.980324 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:45.980185    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj\" (UID: \"adef1f71-a5b9-4715-b3d8-f1c80bda8d8f\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj"
Oct 30 01:35:45.980785 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:45.980756    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj\" (UID: \"adef1f71-a5b9-4715-b3d8-f1c80bda8d8f\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj"
Oct 30 01:35:45.980785 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:45.980769    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj\" (UID: \"adef1f71-a5b9-4715-b3d8-f1c80bda8d8f\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj"
Oct 30 01:35:45.982542 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:45.982522    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj\" (UID: \"adef1f71-a5b9-4715-b3d8-f1c80bda8d8f\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj"
Oct 30 01:35:46.023091 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:46.022926    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmbmw\" (UniqueName: \"kubernetes.io/projected/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-kube-api-access-bmbmw\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj\" (UID: \"adef1f71-a5b9-4715-b3d8-f1c80bda8d8f\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj"
Oct 30 01:35:46.119220 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:46.116341    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj"
Oct 30 01:35:46.261261 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:46.261165    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9" event={"ID":"916074d0-6586-417a-baf6-df529fdd58b8","Type":"ContainerStarted","Data":"e3538048cdbc8fd90adf9b4c57df2858210a5727b63ea545bee0282d9b172671"}
Oct 30 01:35:46.261261 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:46.261223    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9" event={"ID":"916074d0-6586-417a-baf6-df529fdd58b8","Type":"ContainerStarted","Data":"b57d88f2593d19490d0759cff10f72eaa388765f9b0366b77d7ff4e0639e5365"}
Oct 30 01:35:46.306926 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:46.306860    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9" podStartSLOduration=2.306842742 podStartE2EDuration="2.306842742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:46.306248868 +0000 UTC m=+3258.644350778" watchObservedRunningTime="2026-10-30 01:35:46.306842742 +0000 UTC m=+3258.644944652"
Oct 30 01:35:46.642807 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:46.642742    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj"]
Oct 30 01:35:46.974845 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:46.974813    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:35:47.271061 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:47.270964    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj" event={"ID":"adef1f71-a5b9-4715-b3d8-f1c80bda8d8f","Type":"ContainerStarted","Data":"3cb5bf09681c88276de5d79ece6f59421302ffed8be2415da378fc3bceeea98a"}
Oct 30 01:35:47.271061 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:47.271032    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj" event={"ID":"adef1f71-a5b9-4715-b3d8-f1c80bda8d8f","Type":"ContainerStarted","Data":"0d7599c85b0ef837008d484c4dcc5d9c667b3fafc53e619790f142210967270e"}
Oct 30 01:35:47.312298 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:47.312248    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj" podStartSLOduration=2.312232272 podStartE2EDuration="2.312232272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:47.309556807 +0000 UTC m=+3259.647658717" watchObservedRunningTime="2026-10-30 01:35:47.312232272 +0000 UTC m=+3259.650334217"
Oct 30 01:35:48.277697 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:48.277661    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj"
Oct 30 01:35:49.043303 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:35:49.043266    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:35:54.974248 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:54.974209    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:35:54.987035 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:54.987007    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:35:55.341182 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:55.341106    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:35:58.947121 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:58.947083    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2"]
Oct 30 01:35:59.287803 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:59.287768    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj"
Oct 30 01:35:59.360492 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:35:59.360449    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22"]
Oct 30 01:36:03.948398 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:03.948350    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2" podUID="54c7f591-fa59-4890-a139-240501e03109" containerName="payload-processing" containerID="cri-o://29703e2391963a9c9ee6de0748f99cb516870a2dbaba05dae18c8e68dd0bec77" gracePeriod=25
Oct 30 01:36:04.299181 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.299152    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2"
Oct 30 01:36:04.384173 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.380154    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22" podUID="80dbf21d-7985-47dc-8b73-2a13e0a0399c" containerName="payload-processing" containerID="cri-o://4aece0cac57f0d7c23fed9758aec0eb9195e523c4290d13ad62211f023118011" gracePeriod=25
Oct 30 01:36:04.421112 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.420680    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54c7f591-fa59-4890-a139-240501e03109-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c7f591-fa59-4890-a139-240501e03109-plugins-config-volume\") pod \"54c7f591-fa59-4890-a139-240501e03109\" (UID: \"54c7f591-fa59-4890-a139-240501e03109\") "
Oct 30 01:36:04.421112 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.420764    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c7f591-fa59-4890-a139-240501e03109-provider-credentials\" (UniqueName: \"kubernetes.io/projected/54c7f591-fa59-4890-a139-240501e03109-provider-credentials\") pod \"54c7f591-fa59-4890-a139-240501e03109\" (UID: \"54c7f591-fa59-4890-a139-240501e03109\") "
Oct 30 01:36:04.421112 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.420879    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54c7f591-fa59-4890-a139-240501e03109-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54c7f591-fa59-4890-a139-240501e03109-routing-overlay-volume\") pod \"54c7f591-fa59-4890-a139-240501e03109\" (UID: \"54c7f591-fa59-4890-a139-240501e03109\") "
Oct 30 01:36:04.421112 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.420979    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c7f591-fa59-4890-a139-240501e03109-kube-api-access-ssg6p\" (UniqueName: \"kubernetes.io/projected/54c7f591-fa59-4890-a139-240501e03109-kube-api-access-ssg6p\") pod \"54c7f591-fa59-4890-a139-240501e03109\" (UID: \"54c7f591-fa59-4890-a139-240501e03109\") "
Oct 30 01:36:04.421369 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.421165    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c7f591-fa59-4890-a139-240501e03109-plugins-config-volume" pod "54c7f591-fa59-4890-a139-240501e03109" (UID: "54c7f591-fa59-4890-a139-240501e03109"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:36:04.421369 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.421320    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c7f591-fa59-4890-a139-240501e03109-routing-overlay-volume" pod "54c7f591-fa59-4890-a139-240501e03109" (UID: "54c7f591-fa59-4890-a139-240501e03109"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:36:04.421667 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.421644    4864 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c7f591-fa59-4890-a139-240501e03109-plugins-config-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:36:04.422004 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.421669    4864 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54c7f591-fa59-4890-a139-240501e03109-routing-overlay-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:36:04.423381 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.423357    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c7f591-fa59-4890-a139-240501e03109-kube-api-access-ssg6p" pod "54c7f591-fa59-4890-a139-240501e03109" (UID: "54c7f591-fa59-4890-a139-240501e03109"). InnerVolumeSpecName "kube-api-access-ssg6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:36:04.423458 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.423416    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c7f591-fa59-4890-a139-240501e03109-provider-credentials" pod "54c7f591-fa59-4890-a139-240501e03109" (UID: "54c7f591-fa59-4890-a139-240501e03109"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:36:04.496941 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.496674    4864 generic.go:356] "Generic (PLEG): container finished" podID="54c7f591-fa59-4890-a139-240501e03109" containerID="29703e2391963a9c9ee6de0748f99cb516870a2dbaba05dae18c8e68dd0bec77" exitCode=0
Oct 30 01:36:04.496941 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.496734    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2"
Oct 30 01:36:04.496941 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.496741    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2" event={"ID":"54c7f591-fa59-4890-a139-240501e03109","Type":"ContainerDied","Data":"29703e2391963a9c9ee6de0748f99cb516870a2dbaba05dae18c8e68dd0bec77"}
Oct 30 01:36:04.496941 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.496806    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2" event={"ID":"54c7f591-fa59-4890-a139-240501e03109","Type":"ContainerDied","Data":"e0eb169a10b1afc01fa4cdc31f16d5f957fa50ea1a55d78d8632a63ba09baa6d"}
Oct 30 01:36:04.496941 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.496829    4864 scope.go:122] "RemoveContainer" containerID="29703e2391963a9c9ee6de0748f99cb516870a2dbaba05dae18c8e68dd0bec77"
Oct 30 01:36:04.508983 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.508551    4864 scope.go:122] "RemoveContainer" containerID="29703e2391963a9c9ee6de0748f99cb516870a2dbaba05dae18c8e68dd0bec77"
Oct 30 01:36:04.508983 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:36:04.508863    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29703e2391963a9c9ee6de0748f99cb516870a2dbaba05dae18c8e68dd0bec77\": container with ID starting with 29703e2391963a9c9ee6de0748f99cb516870a2dbaba05dae18c8e68dd0bec77 not found: ID does not exist" containerID="29703e2391963a9c9ee6de0748f99cb516870a2dbaba05dae18c8e68dd0bec77"
Oct 30 01:36:04.508983 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.508902    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29703e2391963a9c9ee6de0748f99cb516870a2dbaba05dae18c8e68dd0bec77"} err="failed to get container status \"29703e2391963a9c9ee6de0748f99cb516870a2dbaba05dae18c8e68dd0bec77\": rpc error: code = NotFound desc = could not find container \"29703e2391963a9c9ee6de0748f99cb516870a2dbaba05dae18c8e68dd0bec77\": container with ID starting with 29703e2391963a9c9ee6de0748f99cb516870a2dbaba05dae18c8e68dd0bec77 not found: ID does not exist"
Oct 30 01:36:04.523581 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.523553    4864 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54c7f591-fa59-4890-a139-240501e03109-provider-credentials\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:36:04.523581 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.523582    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssg6p\" (UniqueName: \"kubernetes.io/projected/54c7f591-fa59-4890-a139-240501e03109-kube-api-access-ssg6p\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:36:04.545532 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.545495    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2"]
Oct 30 01:36:04.551375 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.551346    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-6d57wc2b2"]
Oct 30 01:36:04.695663 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.695636    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22"
Oct 30 01:36:04.826513 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.826430    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80dbf21d-7985-47dc-8b73-2a13e0a0399c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/80dbf21d-7985-47dc-8b73-2a13e0a0399c-provider-credentials\") pod \"80dbf21d-7985-47dc-8b73-2a13e0a0399c\" (UID: \"80dbf21d-7985-47dc-8b73-2a13e0a0399c\") "
Oct 30 01:36:04.826513 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.826480    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80dbf21d-7985-47dc-8b73-2a13e0a0399c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80dbf21d-7985-47dc-8b73-2a13e0a0399c-plugins-config-volume\") pod \"80dbf21d-7985-47dc-8b73-2a13e0a0399c\" (UID: \"80dbf21d-7985-47dc-8b73-2a13e0a0399c\") "
Oct 30 01:36:04.826755 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.826574    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80dbf21d-7985-47dc-8b73-2a13e0a0399c-kube-api-access-84wzq\" (UniqueName: \"kubernetes.io/projected/80dbf21d-7985-47dc-8b73-2a13e0a0399c-kube-api-access-84wzq\") pod \"80dbf21d-7985-47dc-8b73-2a13e0a0399c\" (UID: \"80dbf21d-7985-47dc-8b73-2a13e0a0399c\") "
Oct 30 01:36:04.826949 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.826913    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80dbf21d-7985-47dc-8b73-2a13e0a0399c-plugins-config-volume" pod "80dbf21d-7985-47dc-8b73-2a13e0a0399c" (UID: "80dbf21d-7985-47dc-8b73-2a13e0a0399c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:36:04.826949 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.826941    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80dbf21d-7985-47dc-8b73-2a13e0a0399c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80dbf21d-7985-47dc-8b73-2a13e0a0399c-routing-overlay-volume\") pod \"80dbf21d-7985-47dc-8b73-2a13e0a0399c\" (UID: \"80dbf21d-7985-47dc-8b73-2a13e0a0399c\") "
Oct 30 01:36:04.827742 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.827298    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80dbf21d-7985-47dc-8b73-2a13e0a0399c-routing-overlay-volume" pod "80dbf21d-7985-47dc-8b73-2a13e0a0399c" (UID: "80dbf21d-7985-47dc-8b73-2a13e0a0399c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:36:04.827914 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.827895    4864 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80dbf21d-7985-47dc-8b73-2a13e0a0399c-routing-overlay-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:36:04.827914 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.827914    4864 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80dbf21d-7985-47dc-8b73-2a13e0a0399c-plugins-config-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:36:04.828987 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.828967    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80dbf21d-7985-47dc-8b73-2a13e0a0399c-kube-api-access-84wzq" pod "80dbf21d-7985-47dc-8b73-2a13e0a0399c" (UID: "80dbf21d-7985-47dc-8b73-2a13e0a0399c"). InnerVolumeSpecName "kube-api-access-84wzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:36:04.829049 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.828988    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80dbf21d-7985-47dc-8b73-2a13e0a0399c-provider-credentials" pod "80dbf21d-7985-47dc-8b73-2a13e0a0399c" (UID: "80dbf21d-7985-47dc-8b73-2a13e0a0399c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:36:04.929817 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.929782    4864 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80dbf21d-7985-47dc-8b73-2a13e0a0399c-provider-credentials\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:36:04.929817 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:04.929814    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84wzq\" (UniqueName: \"kubernetes.io/projected/80dbf21d-7985-47dc-8b73-2a13e0a0399c-kube-api-access-84wzq\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:36:05.505257 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:05.505222    4864 generic.go:356] "Generic (PLEG): container finished" podID="80dbf21d-7985-47dc-8b73-2a13e0a0399c" containerID="4aece0cac57f0d7c23fed9758aec0eb9195e523c4290d13ad62211f023118011" exitCode=0
Oct 30 01:36:05.505731 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:05.505277    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22"
Oct 30 01:36:05.505731 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:05.505305    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22" event={"ID":"80dbf21d-7985-47dc-8b73-2a13e0a0399c","Type":"ContainerDied","Data":"4aece0cac57f0d7c23fed9758aec0eb9195e523c4290d13ad62211f023118011"}
Oct 30 01:36:05.505731 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:05.505360    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22" event={"ID":"80dbf21d-7985-47dc-8b73-2a13e0a0399c","Type":"ContainerDied","Data":"adaedd5a2eb0cc3da62639ff811cfca0781d58f0da68e9185e9af65b5319461b"}
Oct 30 01:36:05.505731 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:05.505391    4864 scope.go:122] "RemoveContainer" containerID="4aece0cac57f0d7c23fed9758aec0eb9195e523c4290d13ad62211f023118011"
Oct 30 01:36:05.516665 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:05.516644    4864 scope.go:122] "RemoveContainer" containerID="4aece0cac57f0d7c23fed9758aec0eb9195e523c4290d13ad62211f023118011"
Oct 30 01:36:05.516908 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:36:05.516883    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aece0cac57f0d7c23fed9758aec0eb9195e523c4290d13ad62211f023118011\": container with ID starting with 4aece0cac57f0d7c23fed9758aec0eb9195e523c4290d13ad62211f023118011 not found: ID does not exist" containerID="4aece0cac57f0d7c23fed9758aec0eb9195e523c4290d13ad62211f023118011"
Oct 30 01:36:05.517002 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:05.516917    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aece0cac57f0d7c23fed9758aec0eb9195e523c4290d13ad62211f023118011"} err="failed to get container status \"4aece0cac57f0d7c23fed9758aec0eb9195e523c4290d13ad62211f023118011\": rpc error: code = NotFound desc = could not find container \"4aece0cac57f0d7c23fed9758aec0eb9195e523c4290d13ad62211f023118011\": container with ID starting with 4aece0cac57f0d7c23fed9758aec0eb9195e523c4290d13ad62211f023118011 not found: ID does not exist"
Oct 30 01:36:05.572104 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:05.569300    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22"]
Oct 30 01:36:05.593783 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:05.593749    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-6f847ng22"]
Oct 30 01:36:05.795336 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:05.795251    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54c7f591-fa59-4890-a139-240501e03109" path="/var/lib/kubelet/pods/54c7f591-fa59-4890-a139-240501e03109/volumes"
Oct 30 01:36:05.795976 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:05.795950    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80dbf21d-7985-47dc-8b73-2a13e0a0399c" path="/var/lib/kubelet/pods/80dbf21d-7985-47dc-8b73-2a13e0a0399c/volumes"
Oct 30 01:36:08.115266 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.115225    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"]
Oct 30 01:36:08.116875 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.116844    4864 cpu_manager.go:414] "containerMap: removing container" podUID="80dbf21d-7985-47dc-8b73-2a13e0a0399c" containerName="payload-processing"
Oct 30 01:36:08.117064 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.117039    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80dbf21d-7985-47dc-8b73-2a13e0a0399c" containerName="payload-processing"
Oct 30 01:36:08.117195 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.117118    4864 cpu_manager.go:414] "containerMap: removing container" podUID="54c7f591-fa59-4890-a139-240501e03109" containerName="payload-processing"
Oct 30 01:36:08.117195 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.117128    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54c7f591-fa59-4890-a139-240501e03109" containerName="payload-processing"
Oct 30 01:36:08.117343 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.117327    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="54c7f591-fa59-4890-a139-240501e03109" containerName="payload-processing"
Oct 30 01:36:08.117378 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.117366    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="80dbf21d-7985-47dc-8b73-2a13e0a0399c" containerName="payload-processing"
Oct 30 01:36:08.122576 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.122555    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:36:08.148207 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.148177    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ab0a2c\"/\"kube-root-ca.crt\""
Oct 30 01:36:08.148449 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.148374    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ab0a2c\"/\"test-model-ab0a2c-kserve-self-signed-certs\""
Oct 30 01:36:08.148540 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.148460    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ab0a2c\"/\"openshift-service-ca.crt\""
Oct 30 01:36:08.148540 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.148513    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ab0a2c\"/\"default-dockercfg-jvrsv\""
Oct 30 01:36:08.180096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.156246    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"]
Oct 30 01:36:08.301897 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.301860    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-model-cache\") pod \"test-model-ab0a2c-kserve-85cbb78c6f-66j42\" (UID: \"056d952a-9dde-494f-9504-7e104751184b\") " pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:36:08.302126 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.302098    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-home\") pod \"test-model-ab0a2c-kserve-85cbb78c6f-66j42\" (UID: \"056d952a-9dde-494f-9504-7e104751184b\") " pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:36:08.302261 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.302242    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn8hq\" (UniqueName: \"kubernetes.io/projected/056d952a-9dde-494f-9504-7e104751184b-kube-api-access-rn8hq\") pod \"test-model-ab0a2c-kserve-85cbb78c6f-66j42\" (UID: \"056d952a-9dde-494f-9504-7e104751184b\") " pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:36:08.302379 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.302358    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/056d952a-9dde-494f-9504-7e104751184b-tls-certs\") pod \"test-model-ab0a2c-kserve-85cbb78c6f-66j42\" (UID: \"056d952a-9dde-494f-9504-7e104751184b\") " pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:36:08.302491 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.302475    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-dshm\") pod \"test-model-ab0a2c-kserve-85cbb78c6f-66j42\" (UID: \"056d952a-9dde-494f-9504-7e104751184b\") " pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:36:08.302571 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.302557    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-tmp-dir\") pod \"test-model-ab0a2c-kserve-85cbb78c6f-66j42\" (UID: \"056d952a-9dde-494f-9504-7e104751184b\") " pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:36:08.403924 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.403831    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-home\") pod \"test-model-ab0a2c-kserve-85cbb78c6f-66j42\" (UID: \"056d952a-9dde-494f-9504-7e104751184b\") " pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:36:08.403924 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.403904    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn8hq\" (UniqueName: \"kubernetes.io/projected/056d952a-9dde-494f-9504-7e104751184b-kube-api-access-rn8hq\") pod \"test-model-ab0a2c-kserve-85cbb78c6f-66j42\" (UID: \"056d952a-9dde-494f-9504-7e104751184b\") " pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:36:08.404217 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.403971    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/056d952a-9dde-494f-9504-7e104751184b-tls-certs\") pod \"test-model-ab0a2c-kserve-85cbb78c6f-66j42\" (UID: \"056d952a-9dde-494f-9504-7e104751184b\") " pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:36:08.404217 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.404033    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-dshm\") pod \"test-model-ab0a2c-kserve-85cbb78c6f-66j42\" (UID: \"056d952a-9dde-494f-9504-7e104751184b\") " pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:36:08.404217 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.404108    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-tmp-dir\") pod \"test-model-ab0a2c-kserve-85cbb78c6f-66j42\" (UID: \"056d952a-9dde-494f-9504-7e104751184b\") " pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:36:08.404217 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.404155    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-model-cache\") pod \"test-model-ab0a2c-kserve-85cbb78c6f-66j42\" (UID: \"056d952a-9dde-494f-9504-7e104751184b\") " pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:36:08.404456 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.404433    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-home\") pod \"test-model-ab0a2c-kserve-85cbb78c6f-66j42\" (UID: \"056d952a-9dde-494f-9504-7e104751184b\") " pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:36:08.404601 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.404580    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-model-cache\") pod \"test-model-ab0a2c-kserve-85cbb78c6f-66j42\" (UID: \"056d952a-9dde-494f-9504-7e104751184b\") " pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:36:08.404921 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.404896    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-tmp-dir\") pod \"test-model-ab0a2c-kserve-85cbb78c6f-66j42\" (UID: \"056d952a-9dde-494f-9504-7e104751184b\") " pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:36:08.407254 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.406858    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/056d952a-9dde-494f-9504-7e104751184b-tls-certs\") pod \"test-model-ab0a2c-kserve-85cbb78c6f-66j42\" (UID: \"056d952a-9dde-494f-9504-7e104751184b\") " pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:36:08.407356 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.407267    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-dshm\") pod \"test-model-ab0a2c-kserve-85cbb78c6f-66j42\" (UID: \"056d952a-9dde-494f-9504-7e104751184b\") " pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:36:08.432593 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.432554    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn8hq\" (UniqueName: \"kubernetes.io/projected/056d952a-9dde-494f-9504-7e104751184b-kube-api-access-rn8hq\") pod \"test-model-ab0a2c-kserve-85cbb78c6f-66j42\" (UID: \"056d952a-9dde-494f-9504-7e104751184b\") " pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:36:08.438805 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.438573    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:36:08.940451 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.940418    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj"]
Oct 30 01:36:08.999636 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:08.999481    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"]
Oct 30 01:36:09.805151 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:09.805049    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42" event={"ID":"056d952a-9dde-494f-9504-7e104751184b","Type":"ContainerStarted","Data":"646311bbdbc3ecd1ce50ed0088d43338e0933bebddb63e28d3d915104009f6a3"}
Oct 30 01:36:09.805151 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:09.805119    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42" event={"ID":"056d952a-9dde-494f-9504-7e104751184b","Type":"ContainerStarted","Data":"5ca026abfe3bb10048537147f333def3f72aed98d96b1e71cf16ec68c826b911"}
Oct 30 01:36:09.845357 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:09.845306    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42" podStartSLOduration=1.8452918230000002 podStartE2EDuration="1.845291823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:09.843230544 +0000 UTC m=+3282.181332454" watchObservedRunningTime="2026-10-30 01:36:09.845291823 +0000 UTC m=+3282.183393732"
Oct 30 01:36:10.439833 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:10.439801    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:36:13.839287 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:13.839247    4864 generic.go:356] "Generic (PLEG): container finished" podID="09577058-17aa-411b-bf71-092730f470f2" containerID="56af3c0c24f8558222890a96e15ced297db12fe990addeaf2469802fc725f416" exitCode=0
Oct 30 01:36:13.839770 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:13.839321    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb" event={"ID":"09577058-17aa-411b-bf71-092730f470f2","Type":"ContainerDied","Data":"56af3c0c24f8558222890a96e15ced297db12fe990addeaf2469802fc725f416"}
Oct 30 01:36:13.839884 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:13.839867    4864 scope.go:122] "RemoveContainer" containerID="56af3c0c24f8558222890a96e15ced297db12fe990addeaf2469802fc725f416"
Oct 30 01:36:13.842022 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:13.841991    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:36:13.842158 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:13.842100    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:36:13.842235 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:13.842153    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:36:13.941053 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:13.941000    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj" podUID="adef1f71-a5b9-4715-b3d8-f1c80bda8d8f" containerName="payload-processing" containerID="cri-o://3cb5bf09681c88276de5d79ece6f59421302ffed8be2415da378fc3bceeea98a" gracePeriod=25
Oct 30 01:36:14.260296 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:14.260264    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj"
Oct 30 01:36:14.276969 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:14.276935    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:36:14.357287 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:14.356935    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-plugins-config-volume\") pod \"adef1f71-a5b9-4715-b3d8-f1c80bda8d8f\" (UID: \"adef1f71-a5b9-4715-b3d8-f1c80bda8d8f\") "
Oct 30 01:36:14.358354 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:14.358328    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-kube-api-access-bmbmw\" (UniqueName: \"kubernetes.io/projected/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-kube-api-access-bmbmw\") pod \"adef1f71-a5b9-4715-b3d8-f1c80bda8d8f\" (UID: \"adef1f71-a5b9-4715-b3d8-f1c80bda8d8f\") "
Oct 30 01:36:14.366269 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:14.366223    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-routing-overlay-volume\") pod \"adef1f71-a5b9-4715-b3d8-f1c80bda8d8f\" (UID: \"adef1f71-a5b9-4715-b3d8-f1c80bda8d8f\") "
Oct 30 01:36:14.367113 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:14.367063    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-provider-credentials\") pod \"adef1f71-a5b9-4715-b3d8-f1c80bda8d8f\" (UID: \"adef1f71-a5b9-4715-b3d8-f1c80bda8d8f\") "
Oct 30 01:36:14.418174 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:14.358041    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-plugins-config-volume" pod "adef1f71-a5b9-4715-b3d8-f1c80bda8d8f" (UID: "adef1f71-a5b9-4715-b3d8-f1c80bda8d8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:36:14.434425 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:14.366858    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-routing-overlay-volume" pod "adef1f71-a5b9-4715-b3d8-f1c80bda8d8f" (UID: "adef1f71-a5b9-4715-b3d8-f1c80bda8d8f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:36:14.449964 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:14.449648    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-kube-api-access-bmbmw" pod "adef1f71-a5b9-4715-b3d8-f1c80bda8d8f" (UID: "adef1f71-a5b9-4715-b3d8-f1c80bda8d8f"). InnerVolumeSpecName "kube-api-access-bmbmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:36:14.451348 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:14.451301    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-provider-credentials" pod "adef1f71-a5b9-4715-b3d8-f1c80bda8d8f" (UID: "adef1f71-a5b9-4715-b3d8-f1c80bda8d8f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:36:14.476885 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:14.476839    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmbmw\" (UniqueName: \"kubernetes.io/projected/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-kube-api-access-bmbmw\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:36:14.476885 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:14.476882    4864 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-routing-overlay-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:36:14.477183 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:14.476899    4864 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-provider-credentials\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:36:14.477183 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:14.476913    4864 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f-plugins-config-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:36:14.848921 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:14.848876    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb" event={"ID":"09577058-17aa-411b-bf71-092730f470f2","Type":"ContainerStarted","Data":"074ba6540b2eea261321bd974b7db80c6ea682f640127e77a5e6dfa440002c2f"}
Oct 30 01:36:14.850441 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:14.850412    4864 generic.go:356] "Generic (PLEG): container finished" podID="adef1f71-a5b9-4715-b3d8-f1c80bda8d8f" containerID="3cb5bf09681c88276de5d79ece6f59421302ffed8be2415da378fc3bceeea98a" exitCode=0
Oct 30 01:36:14.850588 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:14.850450    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj"
Oct 30 01:36:14.850588 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:14.850496    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj" event={"ID":"adef1f71-a5b9-4715-b3d8-f1c80bda8d8f","Type":"ContainerDied","Data":"3cb5bf09681c88276de5d79ece6f59421302ffed8be2415da378fc3bceeea98a"}
Oct 30 01:36:14.850588 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:14.850544    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj" event={"ID":"adef1f71-a5b9-4715-b3d8-f1c80bda8d8f","Type":"ContainerDied","Data":"0d7599c85b0ef837008d484c4dcc5d9c667b3fafc53e619790f142210967270e"}
Oct 30 01:36:14.850588 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:14.850567    4864 scope.go:122] "RemoveContainer" containerID="3cb5bf09681c88276de5d79ece6f59421302ffed8be2415da378fc3bceeea98a"
Oct 30 01:36:14.888603 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:14.888567    4864 scope.go:122] "RemoveContainer" containerID="3cb5bf09681c88276de5d79ece6f59421302ffed8be2415da378fc3bceeea98a"
Oct 30 01:36:14.889310 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:36:14.889208    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cb5bf09681c88276de5d79ece6f59421302ffed8be2415da378fc3bceeea98a\": container with ID starting with 3cb5bf09681c88276de5d79ece6f59421302ffed8be2415da378fc3bceeea98a not found: ID does not exist" containerID="3cb5bf09681c88276de5d79ece6f59421302ffed8be2415da378fc3bceeea98a"
Oct 30 01:36:14.889310 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:14.889254    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb5bf09681c88276de5d79ece6f59421302ffed8be2415da378fc3bceeea98a"} err="failed to get container status \"3cb5bf09681c88276de5d79ece6f59421302ffed8be2415da378fc3bceeea98a\": rpc error: code = NotFound desc = could not find container \"3cb5bf09681c88276de5d79ece6f59421302ffed8be2415da378fc3bceeea98a\": container with ID starting with 3cb5bf09681c88276de5d79ece6f59421302ffed8be2415da378fc3bceeea98a not found: ID does not exist"
Oct 30 01:36:14.991603 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:14.991565    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj"]
Oct 30 01:36:14.995601 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:14.995570    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9dzqjdj"]
Oct 30 01:36:15.082672 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:15.080431    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:36:15.087092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:15.084177    4864 patch_prober.go:29] interesting pod/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 01:36:15.087092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:15.084236    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb" podUID="09577058-17aa-411b-bf71-092730f470f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 01:36:15.797643 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:15.797606    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adef1f71-a5b9-4715-b3d8-f1c80bda8d8f" path="/var/lib/kubelet/pods/adef1f71-a5b9-4715-b3d8-f1c80bda8d8f/volumes"
Oct 30 01:36:16.078271 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:16.078166    4864 patch_prober.go:29] interesting pod/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 01:36:16.078845 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:16.078808    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb" podUID="09577058-17aa-411b-bf71-092730f470f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 01:36:17.082674 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:17.082628    4864 patch_prober.go:29] interesting pod/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 01:36:17.083282 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:17.083246    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb" podUID="09577058-17aa-411b-bf71-092730f470f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 01:36:18.077914 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:18.077451    4864 patch_prober.go:29] interesting pod/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 01:36:18.077914 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:18.077515    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb" podUID="09577058-17aa-411b-bf71-092730f470f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 01:36:18.440697 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:18.440602    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:36:18.453943 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:18.453919    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:36:19.011749 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:19.011717    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:36:19.043817 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:36:19.043785    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:36:19.085225 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:19.085179    4864 patch_prober.go:29] interesting pod/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 01:36:19.085427 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:19.085259    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb" podUID="09577058-17aa-411b-bf71-092730f470f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 01:36:20.081101 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:20.078815    4864 patch_prober.go:29] interesting pod/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 01:36:20.081101 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:20.078875    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb" podUID="09577058-17aa-411b-bf71-092730f470f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 01:36:21.082786 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:21.082754    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:36:21.083192 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:21.083176    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:36:21.084165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:21.084066    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:36:23.872989 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:23.872953    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc"]
Oct 30 01:36:23.873650 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:23.873635    4864 cpu_manager.go:414] "containerMap: removing container" podUID="adef1f71-a5b9-4715-b3d8-f1c80bda8d8f" containerName="payload-processing"
Oct 30 01:36:23.873695 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:23.873651    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adef1f71-a5b9-4715-b3d8-f1c80bda8d8f" containerName="payload-processing"
Oct 30 01:36:23.873794 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:23.873785    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="adef1f71-a5b9-4715-b3d8-f1c80bda8d8f" containerName="payload-processing"
Oct 30 01:36:23.877222 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:23.877200    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc"
Oct 30 01:36:23.896526 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:23.896499    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc"]
Oct 30 01:36:23.902019 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:23.901989    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6aa9aa\""
Oct 30 01:36:23.949132 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:23.949095    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6"]
Oct 30 01:36:23.973399 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:23.973368    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6"
Oct 30 01:36:24.042763 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:24.014907    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6aa9aa-dockercfg-2xdcq\""
Oct 30 01:36:24.042763 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:24.025758    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8lj9\" (UniqueName: \"kubernetes.io/projected/3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48-kube-api-access-n8lj9\") pod \"payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc\" (UID: \"3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc"
Oct 30 01:36:24.042763 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:24.025848    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc\" (UID: \"3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc"
Oct 30 01:36:24.127166 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:24.093451    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6"]
Oct 30 01:36:24.330828 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:24.326086    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8lj9\" (UniqueName: \"kubernetes.io/projected/3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48-kube-api-access-n8lj9\") pod \"payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc\" (UID: \"3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc"
Oct 30 01:36:24.330828 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:24.326478    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc\" (UID: \"3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc"
Oct 30 01:36:24.330828 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:24.326636    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mngdz\" (UniqueName: \"kubernetes.io/projected/e3d5cee2-a8bf-4694-b410-48282432dc90-kube-api-access-mngdz\") pod \"payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6\" (UID: \"e3d5cee2-a8bf-4694-b410-48282432dc90\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6"
Oct 30 01:36:24.330828 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:24.327026    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3d5cee2-a8bf-4694-b410-48282432dc90-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6\" (UID: \"e3d5cee2-a8bf-4694-b410-48282432dc90\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6"
Oct 30 01:36:24.331866 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:24.331839    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc\" (UID: \"3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc"
Oct 30 01:36:24.352280 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:24.352243    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8lj9\" (UniqueName: \"kubernetes.io/projected/3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48-kube-api-access-n8lj9\") pod \"payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc\" (UID: \"3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc"
Oct 30 01:36:24.430739 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:24.430562    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mngdz\" (UniqueName: \"kubernetes.io/projected/e3d5cee2-a8bf-4694-b410-48282432dc90-kube-api-access-mngdz\") pod \"payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6\" (UID: \"e3d5cee2-a8bf-4694-b410-48282432dc90\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6"
Oct 30 01:36:24.430904 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:24.430778    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3d5cee2-a8bf-4694-b410-48282432dc90-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6\" (UID: \"e3d5cee2-a8bf-4694-b410-48282432dc90\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6"
Oct 30 01:36:24.431536 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:24.431515    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3d5cee2-a8bf-4694-b410-48282432dc90-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6\" (UID: \"e3d5cee2-a8bf-4694-b410-48282432dc90\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6"
Oct 30 01:36:24.452746 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:24.452720    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mngdz\" (UniqueName: \"kubernetes.io/projected/e3d5cee2-a8bf-4694-b410-48282432dc90-kube-api-access-mngdz\") pod \"payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6\" (UID: \"e3d5cee2-a8bf-4694-b410-48282432dc90\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6"
Oct 30 01:36:24.490521 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:24.490475    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc"
Oct 30 01:36:24.615373 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:24.615336    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6"
Oct 30 01:36:24.802192 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:24.802162    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc"]
Oct 30 01:36:24.803300 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:36:24.803277    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0430bb_a6d4_4cad_8a3b_8a5f23ed2e48.slice/crio-98cc2a530ac70395dda5cf02a359307aa7f550e00265e1db57dca87c2cdbfdba WatchSource:0}: Error finding container 98cc2a530ac70395dda5cf02a359307aa7f550e00265e1db57dca87c2cdbfdba: Status 404 returned error can't find the container with id 98cc2a530ac70395dda5cf02a359307aa7f550e00265e1db57dca87c2cdbfdba
Oct 30 01:36:24.916897 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:24.916867    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6"]
Oct 30 01:36:24.917854 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:36:24.917828    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d5cee2_a8bf_4694_b410_48282432dc90.slice/crio-ae48367ff11b26ee40dce95c84bc1dca7a02ba06a3207c82b2cf5e1527c4da1e WatchSource:0}: Error finding container ae48367ff11b26ee40dce95c84bc1dca7a02ba06a3207c82b2cf5e1527c4da1e: Status 404 returned error can't find the container with id ae48367ff11b26ee40dce95c84bc1dca7a02ba06a3207c82b2cf5e1527c4da1e
Oct 30 01:36:25.340128 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:25.340087    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc" event={"ID":"3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48","Type":"ContainerStarted","Data":"63b7117079445d8d12e30dd3e6a5a644210a14fb03848cc7b053de2da9317005"}
Oct 30 01:36:25.340362 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:25.340157    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc" event={"ID":"3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48","Type":"ContainerStarted","Data":"98cc2a530ac70395dda5cf02a359307aa7f550e00265e1db57dca87c2cdbfdba"}
Oct 30 01:36:25.340362 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:25.340187    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc"
Oct 30 01:36:25.341893 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:25.341862    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6" event={"ID":"e3d5cee2-a8bf-4694-b410-48282432dc90","Type":"ContainerStarted","Data":"0fe642ce29d744c4da369343d0eb447051b19cdcac27d9c536bab09ec8068ba8"}
Oct 30 01:36:25.342026 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:25.341915    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6"
Oct 30 01:36:25.342026 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:25.341935    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6" event={"ID":"e3d5cee2-a8bf-4694-b410-48282432dc90","Type":"ContainerStarted","Data":"ae48367ff11b26ee40dce95c84bc1dca7a02ba06a3207c82b2cf5e1527c4da1e"}
Oct 30 01:36:25.378366 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:25.378308    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc" podStartSLOduration=2.378295666 podStartE2EDuration="2.378295666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:25.376867851 +0000 UTC m=+3297.714969784" watchObservedRunningTime="2026-10-30 01:36:25.378295666 +0000 UTC m=+3297.716397612"
Oct 30 01:36:25.413911 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:25.413853    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6" podStartSLOduration=2.413838204 podStartE2EDuration="2.413838204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:25.412999895 +0000 UTC m=+3297.751101799" watchObservedRunningTime="2026-10-30 01:36:25.413838204 +0000 UTC m=+3297.751940114"
Oct 30 01:36:27.941843 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:27.941807    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"]
Oct 30 01:36:27.946501 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:27.946478    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:27.970182 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:27.970155    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-73423b-openshift-default-dockercfg-q8qtg\""
Oct 30 01:36:27.976794 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:27.976747    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"]
Oct 30 01:36:27.986953 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:27.985979    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-credential-socket\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:27.986953 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:27.986058    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg7jz\" (UniqueName: \"kubernetes.io/projected/b6897e74-0146-4ff4-89e3-4f1655fd8f35-kube-api-access-rg7jz\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:27.986953 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:27.986188    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-token\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:27.986953 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:27.986245    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-workload-socket\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:27.986953 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:27.986271    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-data\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:27.986953 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:27.986423    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istiod-ca-cert\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:27.987382 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:27.987121    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-podinfo\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:27.987382 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:27.987210    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-envoy\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:27.987382 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:27.987252    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-workload-certs\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:28.137097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.133706    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-envoy\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:28.137097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.133783    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-workload-certs\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:28.137097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.133924    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-credential-socket\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:28.137097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.133969    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg7jz\" (UniqueName: \"kubernetes.io/projected/b6897e74-0146-4ff4-89e3-4f1655fd8f35-kube-api-access-rg7jz\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:28.137097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.134053    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-token\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:28.137097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.134111    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-workload-socket\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:28.137097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.134134    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-data\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:28.137097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.134211    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istiod-ca-cert\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:28.137097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.134303    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-podinfo\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:28.137097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.136040    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-workload-certs\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:28.137097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.136328    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-workload-socket\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:28.137097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.136816    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-data\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:28.137097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.137040    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-credential-socket\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:28.146108 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.143652    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istiod-ca-cert\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:28.162100 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.160614    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-envoy\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:28.185754 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.183732    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-podinfo\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:28.185754 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.184267    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-token\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:28.221095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.220026    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg7jz\" (UniqueName: \"kubernetes.io/projected/b6897e74-0146-4ff4-89e3-4f1655fd8f35-kube-api-access-rg7jz\") pod \"e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") " pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:28.283092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.277769    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:28.475951 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.475926    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:36:28.500393 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.500363    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:36:28.734099 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.734049    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"]
Oct 30 01:36:28.734416 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:36:28.734389    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6897e74_0146_4ff4_89e3_4f1655fd8f35.slice/crio-3f6904a58421c3e640a211f5130f7c0345b318800876b44814bf01bf7cf80e19 WatchSource:0}: Error finding container 3f6904a58421c3e640a211f5130f7c0345b318800876b44814bf01bf7cf80e19: Status 404 returned error can't find the container with id 3f6904a58421c3e640a211f5130f7c0345b318800876b44814bf01bf7cf80e19
Oct 30 01:36:28.737380 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.737345    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:36:28.737454 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.737434    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:36:28.737499 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.737474    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:36:28.925900 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:28.925868    4864 scope.go:122] "RemoveContainer" containerID="6e9afc1954aa7c4dc0a5efa501b2e49a2135ce114e1bf0babbe85c04e6bb2353"
Oct 30 01:36:29.399851 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:29.399816    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf" event={"ID":"b6897e74-0146-4ff4-89e3-4f1655fd8f35","Type":"ContainerStarted","Data":"cecb3050ac1f23091197f927ec5c3a22172c92b96bd97ebb0baf58fcdc12ae4a"}
Oct 30 01:36:29.400482 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:29.399872    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf" event={"ID":"b6897e74-0146-4ff4-89e3-4f1655fd8f35","Type":"ContainerStarted","Data":"3f6904a58421c3e640a211f5130f7c0345b318800876b44814bf01bf7cf80e19"}
Oct 30 01:36:29.468802 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:29.468734    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf" podStartSLOduration=2.468715443 podStartE2EDuration="2.468715443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:29.456797535 +0000 UTC m=+3301.794899457" watchObservedRunningTime="2026-10-30 01:36:29.468715443 +0000 UTC m=+3301.806817380"
Oct 30 01:36:30.287336 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:30.287295    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:30.293404 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:30.293377    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:30.407429 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:30.407400    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:30.408284 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:30.408266    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:36:34.246559 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:34.246527    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd"]
Oct 30 01:36:34.250440 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:34.250418    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd"
Oct 30 01:36:34.271044 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:34.271022    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-73423b\""
Oct 30 01:36:34.287163 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:34.287138    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd"]
Oct 30 01:36:34.302140 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:34.302102    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g"]
Oct 30 01:36:34.305698 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:34.305676    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g"
Oct 30 01:36:34.323655 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:34.323625    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-73423b-dockercfg-j2lpm\""
Oct 30 01:36:34.329404 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:34.329377    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g"]
Oct 30 01:36:34.370285 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:34.370240    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tnw7\" (UniqueName: \"kubernetes.io/projected/92a9c763-b5f0-4932-aa83-83b5afef6de9-kube-api-access-8tnw7\") pod \"payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd\" (UID: \"92a9c763-b5f0-4932-aa83-83b5afef6de9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd"
Oct 30 01:36:34.370795 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:34.370411    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e167ecf0-dd04-4531-9fbb-d985e9fe5132-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g\" (UID: \"e167ecf0-dd04-4531-9fbb-d985e9fe5132\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g"
Oct 30 01:36:34.487095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:34.479359    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a9c763-b5f0-4932-aa83-83b5afef6de9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd\" (UID: \"92a9c763-b5f0-4932-aa83-83b5afef6de9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd"
Oct 30 01:36:34.487095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:34.479510    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hmfr\" (UniqueName: \"kubernetes.io/projected/e167ecf0-dd04-4531-9fbb-d985e9fe5132-kube-api-access-7hmfr\") pod \"payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g\" (UID: \"e167ecf0-dd04-4531-9fbb-d985e9fe5132\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g"
Oct 30 01:36:34.583451 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:34.582194    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e167ecf0-dd04-4531-9fbb-d985e9fe5132-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g\" (UID: \"e167ecf0-dd04-4531-9fbb-d985e9fe5132\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g"
Oct 30 01:36:34.583451 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:34.582354    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a9c763-b5f0-4932-aa83-83b5afef6de9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd\" (UID: \"92a9c763-b5f0-4932-aa83-83b5afef6de9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd"
Oct 30 01:36:34.583451 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:34.582462    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hmfr\" (UniqueName: \"kubernetes.io/projected/e167ecf0-dd04-4531-9fbb-d985e9fe5132-kube-api-access-7hmfr\") pod \"payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g\" (UID: \"e167ecf0-dd04-4531-9fbb-d985e9fe5132\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g"
Oct 30 01:36:34.583451 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:34.582578    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tnw7\" (UniqueName: \"kubernetes.io/projected/92a9c763-b5f0-4932-aa83-83b5afef6de9-kube-api-access-8tnw7\") pod \"payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd\" (UID: \"92a9c763-b5f0-4932-aa83-83b5afef6de9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd"
Oct 30 01:36:34.588150 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:34.583822    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e167ecf0-dd04-4531-9fbb-d985e9fe5132-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g\" (UID: \"e167ecf0-dd04-4531-9fbb-d985e9fe5132\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g"
Oct 30 01:36:34.588150 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:34.584587    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a9c763-b5f0-4932-aa83-83b5afef6de9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd\" (UID: \"92a9c763-b5f0-4932-aa83-83b5afef6de9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd"
Oct 30 01:36:34.632991 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:34.632714    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tnw7\" (UniqueName: \"kubernetes.io/projected/92a9c763-b5f0-4932-aa83-83b5afef6de9-kube-api-access-8tnw7\") pod \"payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd\" (UID: \"92a9c763-b5f0-4932-aa83-83b5afef6de9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd"
Oct 30 01:36:34.643412 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:34.643379    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hmfr\" (UniqueName: \"kubernetes.io/projected/e167ecf0-dd04-4531-9fbb-d985e9fe5132-kube-api-access-7hmfr\") pod \"payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g\" (UID: \"e167ecf0-dd04-4531-9fbb-d985e9fe5132\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g"
Oct 30 01:36:34.864809 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:34.864726    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd"
Oct 30 01:36:34.922208 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:34.922142    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g"
Oct 30 01:36:35.166066 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:35.166036    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd"]
Oct 30 01:36:35.168806 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:36:35.168778    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a9c763_b5f0_4932_aa83_83b5afef6de9.slice/crio-7b8d374fd60bb51ce7dd4b173329338a2c05566376d525b2109715b7ca108759 WatchSource:0}: Error finding container 7b8d374fd60bb51ce7dd4b173329338a2c05566376d525b2109715b7ca108759: Status 404 returned error can't find the container with id 7b8d374fd60bb51ce7dd4b173329338a2c05566376d525b2109715b7ca108759
Oct 30 01:36:35.257593 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:35.257564    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g"]
Oct 30 01:36:35.260528 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:36:35.260486    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode167ecf0_dd04_4531_9fbb_d985e9fe5132.slice/crio-4f5fe1505b80ae34ab79d448209f49d9fec2b8e626e2ccc19ae8d594017f89f3 WatchSource:0}: Error finding container 4f5fe1505b80ae34ab79d448209f49d9fec2b8e626e2ccc19ae8d594017f89f3: Status 404 returned error can't find the container with id 4f5fe1505b80ae34ab79d448209f49d9fec2b8e626e2ccc19ae8d594017f89f3
Oct 30 01:36:35.535532 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:35.535497    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd" event={"ID":"92a9c763-b5f0-4932-aa83-83b5afef6de9","Type":"ContainerStarted","Data":"2d488f058c9fe4c567c3669dd98f65849fe8f2caf1a5053fc5e762e9f8e0ce31"}
Oct 30 01:36:35.535745 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:35.535561    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd" event={"ID":"92a9c763-b5f0-4932-aa83-83b5afef6de9","Type":"ContainerStarted","Data":"7b8d374fd60bb51ce7dd4b173329338a2c05566376d525b2109715b7ca108759"}
Oct 30 01:36:35.535745 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:35.535590    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd"
Oct 30 01:36:35.537459 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:35.537423    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g" event={"ID":"e167ecf0-dd04-4531-9fbb-d985e9fe5132","Type":"ContainerStarted","Data":"d467f7e0ea7569fc472cb99e53fb3ab6e2435cea4eb907d244f32c13c100483f"}
Oct 30 01:36:35.537824 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:35.537477    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g" event={"ID":"e167ecf0-dd04-4531-9fbb-d985e9fe5132","Type":"ContainerStarted","Data":"4f5fe1505b80ae34ab79d448209f49d9fec2b8e626e2ccc19ae8d594017f89f3"}
Oct 30 01:36:35.537824 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:35.537506    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g"
Oct 30 01:36:35.575670 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:35.575609    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd" podStartSLOduration=1.575594524 podStartE2EDuration="1.575594524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:35.571900956 +0000 UTC m=+3307.910002867" watchObservedRunningTime="2026-10-30 01:36:35.575594524 +0000 UTC m=+3307.913696433"
Oct 30 01:36:35.610521 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:35.610463    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g" podStartSLOduration=1.61044659 podStartE2EDuration="1.61044659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:35.609525155 +0000 UTC m=+3307.947627066" watchObservedRunningTime="2026-10-30 01:36:35.61044659 +0000 UTC m=+3307.948548501"
Oct 30 01:36:36.355406 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:36.355007    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc"
Oct 30 01:36:36.355406 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:36.355287    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6"
Oct 30 01:36:46.555207 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:46.555155    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd"
Oct 30 01:36:46.588906 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:36:46.588873    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g"
Oct 30 01:36:49.061362 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:36:49.061322    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:37:19.046996 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:37:19.046957    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:37:49.042463 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:37:49.042380    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:38:09.540208 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:38:09.540114    4864 generic.go:356] "Generic (PLEG): container finished" podID="b6897e74-0146-4ff4-89e3-4f1655fd8f35" containerID="cecb3050ac1f23091197f927ec5c3a22172c92b96bd97ebb0baf58fcdc12ae4a" exitCode=0
Oct 30 01:38:09.540663 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:38:09.540193    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf" event={"ID":"b6897e74-0146-4ff4-89e3-4f1655fd8f35","Type":"ContainerDied","Data":"cecb3050ac1f23091197f927ec5c3a22172c92b96bd97ebb0baf58fcdc12ae4a"}
Oct 30 01:38:09.540740 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:38:09.540683    4864 scope.go:122] "RemoveContainer" containerID="cecb3050ac1f23091197f927ec5c3a22172c92b96bd97ebb0baf58fcdc12ae4a"
Oct 30 01:38:09.542719 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:38:09.542692    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:38:09.542817 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:38:09.542760    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:38:09.542817 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:38:09.542792    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:38:10.549402 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:38:10.549369    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf" event={"ID":"b6897e74-0146-4ff4-89e3-4f1655fd8f35","Type":"ContainerStarted","Data":"94b1b4d9aa274b8148435ec9dab9997eb6b5bad0140349ab75555a3ca87be763"}
Oct 30 01:38:11.286933 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:38:11.286889    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:38:11.287962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:38:11.287937    4864 patch_prober.go:29] interesting pod/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 01:38:11.288031 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:38:11.288000    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf" podUID="b6897e74-0146-4ff4-89e3-4f1655fd8f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 01:38:12.287198 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:38:12.287168    4864 patch_prober.go:29] interesting pod/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 01:38:12.287618 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:38:12.287232    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf" podUID="b6897e74-0146-4ff4-89e3-4f1655fd8f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 01:38:13.287328 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:38:13.287288    4864 patch_prober.go:29] interesting pod/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 01:38:13.287816 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:38:13.287356    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf" podUID="b6897e74-0146-4ff4-89e3-4f1655fd8f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 01:38:14.287234 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:38:14.287200    4864 patch_prober.go:29] interesting pod/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 01:38:14.287423 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:38:14.287258    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf" podUID="b6897e74-0146-4ff4-89e3-4f1655fd8f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 01:38:15.287323 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:38:15.287287    4864 patch_prober.go:29] interesting pod/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 01:38:15.287528 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:38:15.287353    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf" podUID="b6897e74-0146-4ff4-89e3-4f1655fd8f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 01:38:15.407881 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:38:15.407848    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:38:16.287178 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:38:16.287139    4864 patch_prober.go:29] interesting pod/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 01:38:16.287370 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:38:16.287197    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf" podUID="b6897e74-0146-4ff4-89e3-4f1655fd8f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 01:38:17.292635 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:38:17.292603    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:38:17.293718 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:38:17.293695    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:38:19.048852 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:38:19.048819    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:38:49.041656 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:38:49.041617    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:39:06.419418 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:06.419380    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"]
Oct 30 01:39:06.419849 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:06.419671    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j" podUID="01101e66-e714-4a99-af87-3e8e0e895867" containerName="istio-proxy" containerID="cri-o://804d68c1125a5defc4e3db4f0c20fe1f40192d451cd0e62af663dc305ff9bfba" gracePeriod=30
Oct 30 01:39:10.349545 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.349463    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"]
Oct 30 01:39:10.354150 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.354127    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:39:10.369936 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.369910    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b29d4a\"/\"default-dockercfg-pk42r\""
Oct 30 01:39:10.380412 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.380384    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b29d4a\"/\"kube-root-ca.crt\""
Oct 30 01:39:10.380589 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.380410    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b29d4a\"/\"openshift-service-ca.crt\""
Oct 30 01:39:10.380681 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.380437    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b29d4a\"/\"test-model-b29d4a-kserve-self-signed-certs\""
Oct 30 01:39:10.387209 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.387183    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"]
Oct 30 01:39:10.435053 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.435019    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a2e5ac-0716-4f26-bc37-b0682e5c8988-tls-certs\") pod \"test-model-b29d4a-kserve-56c88dd54c-km2lb\" (UID: \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\") " pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:39:10.435221 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.435185    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-home\") pod \"test-model-b29d4a-kserve-56c88dd54c-km2lb\" (UID: \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\") " pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:39:10.435288 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.435257    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-dshm\") pod \"test-model-b29d4a-kserve-56c88dd54c-km2lb\" (UID: \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\") " pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:39:10.435379 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.435355    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px6gl\" (UniqueName: \"kubernetes.io/projected/20a2e5ac-0716-4f26-bc37-b0682e5c8988-kube-api-access-px6gl\") pod \"test-model-b29d4a-kserve-56c88dd54c-km2lb\" (UID: \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\") " pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:39:10.435490 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.435467    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-tmp-dir\") pod \"test-model-b29d4a-kserve-56c88dd54c-km2lb\" (UID: \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\") " pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:39:10.435565 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.435553    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-model-cache\") pod \"test-model-b29d4a-kserve-56c88dd54c-km2lb\" (UID: \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\") " pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:39:10.536490 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.536454    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-home\") pod \"test-model-b29d4a-kserve-56c88dd54c-km2lb\" (UID: \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\") " pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:39:10.536654 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.536549    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-dshm\") pod \"test-model-b29d4a-kserve-56c88dd54c-km2lb\" (UID: \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\") " pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:39:10.536654 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.536581    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px6gl\" (UniqueName: \"kubernetes.io/projected/20a2e5ac-0716-4f26-bc37-b0682e5c8988-kube-api-access-px6gl\") pod \"test-model-b29d4a-kserve-56c88dd54c-km2lb\" (UID: \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\") " pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:39:10.536727 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.536650    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-tmp-dir\") pod \"test-model-b29d4a-kserve-56c88dd54c-km2lb\" (UID: \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\") " pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:39:10.536781 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.536754    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-model-cache\") pod \"test-model-b29d4a-kserve-56c88dd54c-km2lb\" (UID: \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\") " pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:39:10.536869 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.536846    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a2e5ac-0716-4f26-bc37-b0682e5c8988-tls-certs\") pod \"test-model-b29d4a-kserve-56c88dd54c-km2lb\" (UID: \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\") " pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:39:10.536926 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.536889    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-home\") pod \"test-model-b29d4a-kserve-56c88dd54c-km2lb\" (UID: \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\") " pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:39:10.537175 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.537147    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-tmp-dir\") pod \"test-model-b29d4a-kserve-56c88dd54c-km2lb\" (UID: \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\") " pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:39:10.537313 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.537227    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-model-cache\") pod \"test-model-b29d4a-kserve-56c88dd54c-km2lb\" (UID: \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\") " pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:39:10.539156 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.539133    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-dshm\") pod \"test-model-b29d4a-kserve-56c88dd54c-km2lb\" (UID: \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\") " pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:39:10.539370 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.539356    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a2e5ac-0716-4f26-bc37-b0682e5c8988-tls-certs\") pod \"test-model-b29d4a-kserve-56c88dd54c-km2lb\" (UID: \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\") " pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:39:10.558218 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.558185    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px6gl\" (UniqueName: \"kubernetes.io/projected/20a2e5ac-0716-4f26-bc37-b0682e5c8988-kube-api-access-px6gl\") pod \"test-model-b29d4a-kserve-56c88dd54c-km2lb\" (UID: \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\") " pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:39:10.672461 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:10.672363    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:39:11.061419 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.061389    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"]
Oct 30 01:39:11.063237 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:39:11.063206    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20a2e5ac_0716_4f26_bc37_b0682e5c8988.slice/crio-1edd8731812703280f457a63dd0ee704623059b6b58442cac6346d9bfd724434 WatchSource:0}: Error finding container 1edd8731812703280f457a63dd0ee704623059b6b58442cac6346d9bfd724434: Status 404 returned error can't find the container with id 1edd8731812703280f457a63dd0ee704623059b6b58442cac6346d9bfd724434
Oct 30 01:39:11.130056 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.130027    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb" event={"ID":"20a2e5ac-0716-4f26-bc37-b0682e5c8988","Type":"ContainerStarted","Data":"1edd8731812703280f457a63dd0ee704623059b6b58442cac6346d9bfd724434"}
Oct 30 01:39:11.708009 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.707977    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:39:11.764835 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.764803    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-istio-envoy\") pod \"01101e66-e714-4a99-af87-3e8e0e895867\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") "
Oct 30 01:39:11.765027 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.764931    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-workload-socket\") pod \"01101e66-e714-4a99-af87-3e8e0e895867\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") "
Oct 30 01:39:11.765027 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.765016    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01101e66-e714-4a99-af87-3e8e0e895867-kube-api-access-rfnhq\" (UniqueName: \"kubernetes.io/projected/01101e66-e714-4a99-af87-3e8e0e895867-kube-api-access-rfnhq\") pod \"01101e66-e714-4a99-af87-3e8e0e895867\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") "
Oct 30 01:39:11.765204 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.765119    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01101e66-e714-4a99-af87-3e8e0e895867-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01101e66-e714-4a99-af87-3e8e0e895867-istio-podinfo\") pod \"01101e66-e714-4a99-af87-3e8e0e895867\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") "
Oct 30 01:39:11.765204 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.765163    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-istio-data\") pod \"01101e66-e714-4a99-af87-3e8e0e895867\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") "
Oct 30 01:39:11.765326 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.765218    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01101e66-e714-4a99-af87-3e8e0e895867-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01101e66-e714-4a99-af87-3e8e0e895867-istiod-ca-cert\") pod \"01101e66-e714-4a99-af87-3e8e0e895867\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") "
Oct 30 01:39:11.765326 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.765256    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-workload-certs\") pod \"01101e66-e714-4a99-af87-3e8e0e895867\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") "
Oct 30 01:39:11.765326 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.765262    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-workload-socket" pod "01101e66-e714-4a99-af87-3e8e0e895867" (UID: "01101e66-e714-4a99-af87-3e8e0e895867"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:39:11.765481 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.765440    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-credential-socket\") pod \"01101e66-e714-4a99-af87-3e8e0e895867\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") "
Oct 30 01:39:11.765668 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.765644    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01101e66-e714-4a99-af87-3e8e0e895867-istio-token\" (UniqueName: \"kubernetes.io/projected/01101e66-e714-4a99-af87-3e8e0e895867-istio-token\") pod \"01101e66-e714-4a99-af87-3e8e0e895867\" (UID: \"01101e66-e714-4a99-af87-3e8e0e895867\") "
Oct 30 01:39:11.766470 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.766438    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:39:11.767210 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.767183    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-workload-certs" pod "01101e66-e714-4a99-af87-3e8e0e895867" (UID: "01101e66-e714-4a99-af87-3e8e0e895867"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:39:11.767403 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.767383    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-credential-socket" pod "01101e66-e714-4a99-af87-3e8e0e895867" (UID: "01101e66-e714-4a99-af87-3e8e0e895867"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:39:11.767811 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.767784    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01101e66-e714-4a99-af87-3e8e0e895867-istiod-ca-cert" pod "01101e66-e714-4a99-af87-3e8e0e895867" (UID: "01101e66-e714-4a99-af87-3e8e0e895867"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:39:11.767811 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.767795    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-istio-data" pod "01101e66-e714-4a99-af87-3e8e0e895867" (UID: "01101e66-e714-4a99-af87-3e8e0e895867"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:39:11.768712 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.768685    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-istio-envoy" pod "01101e66-e714-4a99-af87-3e8e0e895867" (UID: "01101e66-e714-4a99-af87-3e8e0e895867"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:39:11.769301 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.769277    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01101e66-e714-4a99-af87-3e8e0e895867-kube-api-access-rfnhq" pod "01101e66-e714-4a99-af87-3e8e0e895867" (UID: "01101e66-e714-4a99-af87-3e8e0e895867"). InnerVolumeSpecName "kube-api-access-rfnhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:39:11.769593 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.769556    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01101e66-e714-4a99-af87-3e8e0e895867-istio-podinfo" pod "01101e66-e714-4a99-af87-3e8e0e895867" (UID: "01101e66-e714-4a99-af87-3e8e0e895867"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:39:11.770255 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.769897    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01101e66-e714-4a99-af87-3e8e0e895867-istio-token" pod "01101e66-e714-4a99-af87-3e8e0e895867" (UID: "01101e66-e714-4a99-af87-3e8e0e895867"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:39:11.867840 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.867807    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:39:11.867840 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.867838    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01101e66-e714-4a99-af87-3e8e0e895867-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:39:11.867840 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.867852    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:39:11.868109 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.867865    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfnhq\" (UniqueName: \"kubernetes.io/projected/01101e66-e714-4a99-af87-3e8e0e895867-kube-api-access-rfnhq\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:39:11.868109 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.867878    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01101e66-e714-4a99-af87-3e8e0e895867-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:39:11.868109 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.867892    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:39:11.868109 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.867903    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01101e66-e714-4a99-af87-3e8e0e895867-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:39:11.868109 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:11.867915    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01101e66-e714-4a99-af87-3e8e0e895867-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:39:12.141112 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:12.141002    4864 generic.go:356] "Generic (PLEG): container finished" podID="01101e66-e714-4a99-af87-3e8e0e895867" containerID="804d68c1125a5defc4e3db4f0c20fe1f40192d451cd0e62af663dc305ff9bfba" exitCode=0
Oct 30 01:39:12.141112 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:12.141097    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"
Oct 30 01:39:12.141356 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:12.141123    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j" event={"ID":"01101e66-e714-4a99-af87-3e8e0e895867","Type":"ContainerDied","Data":"804d68c1125a5defc4e3db4f0c20fe1f40192d451cd0e62af663dc305ff9bfba"}
Oct 30 01:39:12.141356 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:12.141196    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j" event={"ID":"01101e66-e714-4a99-af87-3e8e0e895867","Type":"ContainerDied","Data":"fa3441cf72c27a74dc8ee10e6bbfc236ff823e80d9c8320c8aa570f132b5da44"}
Oct 30 01:39:12.141356 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:12.141227    4864 scope.go:122] "RemoveContainer" containerID="804d68c1125a5defc4e3db4f0c20fe1f40192d451cd0e62af663dc305ff9bfba"
Oct 30 01:39:12.144403 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:12.144369    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb" event={"ID":"20a2e5ac-0716-4f26-bc37-b0682e5c8988","Type":"ContainerStarted","Data":"7d926e8f34cda5f906076181841f08172b3f3f3dd69089f8ae60947708e0138a"}
Oct 30 01:39:12.154280 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:12.154219    4864 scope.go:122] "RemoveContainer" containerID="804d68c1125a5defc4e3db4f0c20fe1f40192d451cd0e62af663dc305ff9bfba"
Oct 30 01:39:12.154572 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:39:12.154554    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"804d68c1125a5defc4e3db4f0c20fe1f40192d451cd0e62af663dc305ff9bfba\": container with ID starting with 804d68c1125a5defc4e3db4f0c20fe1f40192d451cd0e62af663dc305ff9bfba not found: ID does not exist" containerID="804d68c1125a5defc4e3db4f0c20fe1f40192d451cd0e62af663dc305ff9bfba"
Oct 30 01:39:12.154649 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:12.154579    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"804d68c1125a5defc4e3db4f0c20fe1f40192d451cd0e62af663dc305ff9bfba"} err="failed to get container status \"804d68c1125a5defc4e3db4f0c20fe1f40192d451cd0e62af663dc305ff9bfba\": rpc error: code = NotFound desc = could not find container \"804d68c1125a5defc4e3db4f0c20fe1f40192d451cd0e62af663dc305ff9bfba\": container with ID starting with 804d68c1125a5defc4e3db4f0c20fe1f40192d451cd0e62af663dc305ff9bfba not found: ID does not exist"
Oct 30 01:39:12.190116 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:12.190067    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"]
Oct 30 01:39:12.200475 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:12.200447    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2h99j"]
Oct 30 01:39:12.238112 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:12.238047    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb" podStartSLOduration=2.238034056 podStartE2EDuration="2.238034056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:12.234243546 +0000 UTC m=+3464.572345457" watchObservedRunningTime="2026-10-30 01:39:12.238034056 +0000 UTC m=+3464.576136008"
Oct 30 01:39:12.673355 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:12.673322    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:39:13.752132 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.752095    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"]
Oct 30 01:39:13.753331 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.753308    4864 cpu_manager.go:414] "containerMap: removing container" podUID="01101e66-e714-4a99-af87-3e8e0e895867" containerName="istio-proxy"
Oct 30 01:39:13.753402 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.753333    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01101e66-e714-4a99-af87-3e8e0e895867" containerName="istio-proxy"
Oct 30 01:39:13.753524 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.753512    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="01101e66-e714-4a99-af87-3e8e0e895867" containerName="istio-proxy"
Oct 30 01:39:13.759049 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.759025    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:39:13.780546 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.780496    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"]
Oct 30 01:39:13.786471 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.786442    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5457f2\"/\"kube-root-ca.crt\""
Oct 30 01:39:13.786471 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.786456    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5457f2\"/\"test-model-5457f2-kserve-self-signed-certs\""
Oct 30 01:39:13.786654 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.786518    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5457f2\"/\"openshift-service-ca.crt\""
Oct 30 01:39:13.786654 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.786599    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5457f2\"/\"default-dockercfg-zrds6\""
Oct 30 01:39:13.798719 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.798691    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01101e66-e714-4a99-af87-3e8e0e895867" path="/var/lib/kubelet/pods/01101e66-e714-4a99-af87-3e8e0e895867/volumes"
Oct 30 01:39:13.809202 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.809171    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-home\") pod \"test-model-5457f2-kserve-98946549-gmzfs\" (UID: \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\") " pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:39:13.809343 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.809289    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-dshm\") pod \"test-model-5457f2-kserve-98946549-gmzfs\" (UID: \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\") " pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:39:13.809343 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.809325    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-tmp-dir\") pod \"test-model-5457f2-kserve-98946549-gmzfs\" (UID: \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\") " pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:39:13.809476 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.809398    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4kr8\" (UniqueName: \"kubernetes.io/projected/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-kube-api-access-s4kr8\") pod \"test-model-5457f2-kserve-98946549-gmzfs\" (UID: \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\") " pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:39:13.809565 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.809548    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-model-cache\") pod \"test-model-5457f2-kserve-98946549-gmzfs\" (UID: \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\") " pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:39:13.809639 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.809579    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-tls-certs\") pod \"test-model-5457f2-kserve-98946549-gmzfs\" (UID: \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\") " pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:39:13.910851 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.910814    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-dshm\") pod \"test-model-5457f2-kserve-98946549-gmzfs\" (UID: \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\") " pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:39:13.910851 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.910851    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-tmp-dir\") pod \"test-model-5457f2-kserve-98946549-gmzfs\" (UID: \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\") " pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:39:13.911182 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.910918    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4kr8\" (UniqueName: \"kubernetes.io/projected/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-kube-api-access-s4kr8\") pod \"test-model-5457f2-kserve-98946549-gmzfs\" (UID: \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\") " pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:39:13.911182 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.911118    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-model-cache\") pod \"test-model-5457f2-kserve-98946549-gmzfs\" (UID: \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\") " pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:39:13.911182 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.911144    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-tls-certs\") pod \"test-model-5457f2-kserve-98946549-gmzfs\" (UID: \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\") " pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:39:13.911364 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.911339    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-home\") pod \"test-model-5457f2-kserve-98946549-gmzfs\" (UID: \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\") " pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:39:13.911475 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.911449    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-model-cache\") pod \"test-model-5457f2-kserve-98946549-gmzfs\" (UID: \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\") " pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:39:13.911548 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.911337    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-tmp-dir\") pod \"test-model-5457f2-kserve-98946549-gmzfs\" (UID: \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\") " pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:39:13.911730 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.911714    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-home\") pod \"test-model-5457f2-kserve-98946549-gmzfs\" (UID: \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\") " pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:39:13.913304 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.913279    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-dshm\") pod \"test-model-5457f2-kserve-98946549-gmzfs\" (UID: \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\") " pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:39:13.913613 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.913596    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-tls-certs\") pod \"test-model-5457f2-kserve-98946549-gmzfs\" (UID: \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\") " pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:39:13.934060 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:13.934033    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4kr8\" (UniqueName: \"kubernetes.io/projected/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-kube-api-access-s4kr8\") pod \"test-model-5457f2-kserve-98946549-gmzfs\" (UID: \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\") " pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:39:14.083141 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:14.076152    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:39:14.462553 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:14.462527    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"]
Oct 30 01:39:14.463252 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:39:14.463227    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28a7413b_beb4_4b9f_bc76_7b1c5497ce1d.slice/crio-1d15dd068ae9533f3aa92e17947131cc249c4cc201c5d92906e90e83413ac068 WatchSource:0}: Error finding container 1d15dd068ae9533f3aa92e17947131cc249c4cc201c5d92906e90e83413ac068: Status 404 returned error can't find the container with id 1d15dd068ae9533f3aa92e17947131cc249c4cc201c5d92906e90e83413ac068
Oct 30 01:39:15.180337 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:15.180297    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs" event={"ID":"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d","Type":"ContainerStarted","Data":"b98fb73783afe0ef43e59d1762ac0e2d688ff51d5f89c1b00c467d8936451faf"}
Oct 30 01:39:15.180728 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:15.180348    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs" event={"ID":"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d","Type":"ContainerStarted","Data":"1d15dd068ae9533f3aa92e17947131cc249c4cc201c5d92906e90e83413ac068"}
Oct 30 01:39:15.219581 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:15.219526    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs" podStartSLOduration=2.2195134850000002 podStartE2EDuration="2.219513485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:15.217823544 +0000 UTC m=+3467.555925446" watchObservedRunningTime="2026-10-30 01:39:15.219513485 +0000 UTC m=+3467.557615394"
Oct 30 01:39:16.087226 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:16.087190    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:39:19.040763 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:39:19.040728    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:39:20.673278 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:20.673247    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:39:20.686415 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:20.686388    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:39:21.236486 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:21.236447    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:39:24.087912 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:24.087872    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:39:24.100961 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:24.100935    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:39:24.261291 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:24.261264    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:39:27.771547 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:27.771507    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"]
Oct 30 01:39:27.772041 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:27.771712    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z" podUID="3aa394cb-eba6-4e15-9a0e-523e2379f7ac" containerName="istio-proxy" containerID="cri-o://ed5252895c5d24c30c3931649ccc0b63aa0f09fff0f0551fbe1a03daa6f62bd0" gracePeriod=30
Oct 30 01:39:31.565656 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:31.565620    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7"]
Oct 30 01:39:31.571717 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:31.571673    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7"
Oct 30 01:39:31.594298 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:31.594261    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1a4b17-serving-cert\""
Oct 30 01:39:31.601600 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:31.601574    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7"]
Oct 30 01:39:31.635062 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:31.635023    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9254f264-647e-42f6-876a-3b38efcd829f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7\" (UID: \"9254f264-647e-42f6-876a-3b38efcd829f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7"
Oct 30 01:39:31.635249 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:31.635166    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9254f264-647e-42f6-876a-3b38efcd829f-metrics-certs\") pod \"maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7\" (UID: \"9254f264-647e-42f6-876a-3b38efcd829f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7"
Oct 30 01:39:31.635303 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:31.635257    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9254f264-647e-42f6-876a-3b38efcd829f-maas-api-tls\") pod \"maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7\" (UID: \"9254f264-647e-42f6-876a-3b38efcd829f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7"
Oct 30 01:39:31.635412 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:31.635393    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr5lw\" (UniqueName: \"kubernetes.io/projected/9254f264-647e-42f6-876a-3b38efcd829f-kube-api-access-dr5lw\") pod \"maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7\" (UID: \"9254f264-647e-42f6-876a-3b38efcd829f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7"
Oct 30 01:39:31.736909 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:31.736856    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9254f264-647e-42f6-876a-3b38efcd829f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7\" (UID: \"9254f264-647e-42f6-876a-3b38efcd829f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7"
Oct 30 01:39:31.737162 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:31.736953    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9254f264-647e-42f6-876a-3b38efcd829f-metrics-certs\") pod \"maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7\" (UID: \"9254f264-647e-42f6-876a-3b38efcd829f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7"
Oct 30 01:39:31.737162 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:31.737016    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9254f264-647e-42f6-876a-3b38efcd829f-maas-api-tls\") pod \"maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7\" (UID: \"9254f264-647e-42f6-876a-3b38efcd829f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7"
Oct 30 01:39:31.737162 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:31.737104    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr5lw\" (UniqueName: \"kubernetes.io/projected/9254f264-647e-42f6-876a-3b38efcd829f-kube-api-access-dr5lw\") pod \"maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7\" (UID: \"9254f264-647e-42f6-876a-3b38efcd829f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7"
Oct 30 01:39:31.737652 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:31.737626    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9254f264-647e-42f6-876a-3b38efcd829f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7\" (UID: \"9254f264-647e-42f6-876a-3b38efcd829f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7"
Oct 30 01:39:31.739544 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:31.739520    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9254f264-647e-42f6-876a-3b38efcd829f-maas-api-tls\") pod \"maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7\" (UID: \"9254f264-647e-42f6-876a-3b38efcd829f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7"
Oct 30 01:39:31.739697 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:31.739633    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9254f264-647e-42f6-876a-3b38efcd829f-metrics-certs\") pod \"maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7\" (UID: \"9254f264-647e-42f6-876a-3b38efcd829f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7"
Oct 30 01:39:31.796987 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:31.796686    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr5lw\" (UniqueName: \"kubernetes.io/projected/9254f264-647e-42f6-876a-3b38efcd829f-kube-api-access-dr5lw\") pod \"maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7\" (UID: \"9254f264-647e-42f6-876a-3b38efcd829f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7"
Oct 30 01:39:31.886316 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:31.886227    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7"
Oct 30 01:39:32.069539 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:32.069501    4864 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:39:32.069707 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:32.069568    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z" podUID="3aa394cb-eba6-4e15-9a0e-523e2379f7ac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:39:32.190857 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:32.190826    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7"]
Oct 30 01:39:32.191777 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:39:32.191750    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9254f264_647e_42f6_876a_3b38efcd829f.slice/crio-db1e42e59fffb63ecd85d2406556999a43a40a9e7a3f074b22317e277d7d23f8 WatchSource:0}: Error finding container db1e42e59fffb63ecd85d2406556999a43a40a9e7a3f074b22317e277d7d23f8: Status 404 returned error can't find the container with id db1e42e59fffb63ecd85d2406556999a43a40a9e7a3f074b22317e277d7d23f8
Oct 30 01:39:32.193361 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:32.193342    4864 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:39:32.313574 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:32.313537    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7" event={"ID":"9254f264-647e-42f6-876a-3b38efcd829f","Type":"ContainerStarted","Data":"db1e42e59fffb63ecd85d2406556999a43a40a9e7a3f074b22317e277d7d23f8"}
Oct 30 01:39:33.122593 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.122563    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:39:33.168417 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.168348    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-token\" (UniqueName: \"kubernetes.io/projected/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-token\") pod \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") "
Oct 30 01:39:33.168564 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.168465    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-envoy\") pod \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") "
Oct 30 01:39:33.168564 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.168516    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istiod-ca-cert\") pod \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") "
Oct 30 01:39:33.168564 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.168541    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-workload-socket\") pod \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") "
Oct 30 01:39:33.168742 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.168600    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-kube-api-access-2xlv2\" (UniqueName: \"kubernetes.io/projected/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-kube-api-access-2xlv2\") pod \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") "
Oct 30 01:39:33.168742 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.168671    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-podinfo\") pod \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") "
Oct 30 01:39:33.168742 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.168697    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-credential-socket\") pod \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") "
Oct 30 01:39:33.168903 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.168816    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-workload-socket" pod "3aa394cb-eba6-4e15-9a0e-523e2379f7ac" (UID: "3aa394cb-eba6-4e15-9a0e-523e2379f7ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:39:33.168903 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.168829    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-workload-certs\") pod \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") "
Oct 30 01:39:33.169024 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.168915    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istiod-ca-cert" pod "3aa394cb-eba6-4e15-9a0e-523e2379f7ac" (UID: "3aa394cb-eba6-4e15-9a0e-523e2379f7ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:39:33.169024 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.168932    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-data\") pod \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\" (UID: \"3aa394cb-eba6-4e15-9a0e-523e2379f7ac\") "
Oct 30 01:39:33.169167 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.169054    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-credential-socket" pod "3aa394cb-eba6-4e15-9a0e-523e2379f7ac" (UID: "3aa394cb-eba6-4e15-9a0e-523e2379f7ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:39:33.169234 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.169162    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-workload-certs" pod "3aa394cb-eba6-4e15-9a0e-523e2379f7ac" (UID: "3aa394cb-eba6-4e15-9a0e-523e2379f7ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:39:33.169318 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.169290    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-data" pod "3aa394cb-eba6-4e15-9a0e-523e2379f7ac" (UID: "3aa394cb-eba6-4e15-9a0e-523e2379f7ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:39:33.169823 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.169796    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:39:33.169931 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.169829    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:39:33.169931 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.169846    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:39:33.169931 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.169861    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:39:33.169931 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.169876    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:39:33.170766 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.170739    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-token" pod "3aa394cb-eba6-4e15-9a0e-523e2379f7ac" (UID: "3aa394cb-eba6-4e15-9a0e-523e2379f7ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:39:33.170876 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.170834    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-envoy" pod "3aa394cb-eba6-4e15-9a0e-523e2379f7ac" (UID: "3aa394cb-eba6-4e15-9a0e-523e2379f7ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:39:33.171023 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.171008    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-podinfo" pod "3aa394cb-eba6-4e15-9a0e-523e2379f7ac" (UID: "3aa394cb-eba6-4e15-9a0e-523e2379f7ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:39:33.171121 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.171100    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-kube-api-access-2xlv2" pod "3aa394cb-eba6-4e15-9a0e-523e2379f7ac" (UID: "3aa394cb-eba6-4e15-9a0e-523e2379f7ac"). InnerVolumeSpecName "kube-api-access-2xlv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:39:33.271150 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.271118    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xlv2\" (UniqueName: \"kubernetes.io/projected/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-kube-api-access-2xlv2\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:39:33.271150 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.271146    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:39:33.271150 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.271156    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:39:33.271386 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.271165    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa394cb-eba6-4e15-9a0e-523e2379f7ac-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:39:33.342094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.337750    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7" event={"ID":"9254f264-647e-42f6-876a-3b38efcd829f","Type":"ContainerStarted","Data":"d515627ca85a88eb50ceb7e92fa43e4bb6f91b557dae2678b26cbc75e0d32f1e"}
Oct 30 01:39:33.342094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.339195    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7"
Oct 30 01:39:33.342094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.341257    4864 generic.go:356] "Generic (PLEG): container finished" podID="3aa394cb-eba6-4e15-9a0e-523e2379f7ac" containerID="ed5252895c5d24c30c3931649ccc0b63aa0f09fff0f0551fbe1a03daa6f62bd0" exitCode=0
Oct 30 01:39:33.342094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.341531    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z" event={"ID":"3aa394cb-eba6-4e15-9a0e-523e2379f7ac","Type":"ContainerDied","Data":"ed5252895c5d24c30c3931649ccc0b63aa0f09fff0f0551fbe1a03daa6f62bd0"}
Oct 30 01:39:33.342094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.341575    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z" event={"ID":"3aa394cb-eba6-4e15-9a0e-523e2379f7ac","Type":"ContainerDied","Data":"6b0d4712ab3d91f752cb3ca6599d1932abb678d1b77936db5f62b9092577d910"}
Oct 30 01:39:33.342094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.341606    4864 scope.go:122] "RemoveContainer" containerID="ed5252895c5d24c30c3931649ccc0b63aa0f09fff0f0551fbe1a03daa6f62bd0"
Oct 30 01:39:33.342094 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.341747    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"
Oct 30 01:39:33.359621 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.359600    4864 scope.go:122] "RemoveContainer" containerID="ed5252895c5d24c30c3931649ccc0b63aa0f09fff0f0551fbe1a03daa6f62bd0"
Oct 30 01:39:33.359921 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:39:33.359899    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed5252895c5d24c30c3931649ccc0b63aa0f09fff0f0551fbe1a03daa6f62bd0\": container with ID starting with ed5252895c5d24c30c3931649ccc0b63aa0f09fff0f0551fbe1a03daa6f62bd0 not found: ID does not exist" containerID="ed5252895c5d24c30c3931649ccc0b63aa0f09fff0f0551fbe1a03daa6f62bd0"
Oct 30 01:39:33.360032 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.359928    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed5252895c5d24c30c3931649ccc0b63aa0f09fff0f0551fbe1a03daa6f62bd0"} err="failed to get container status \"ed5252895c5d24c30c3931649ccc0b63aa0f09fff0f0551fbe1a03daa6f62bd0\": rpc error: code = NotFound desc = could not find container \"ed5252895c5d24c30c3931649ccc0b63aa0f09fff0f0551fbe1a03daa6f62bd0\": container with ID starting with ed5252895c5d24c30c3931649ccc0b63aa0f09fff0f0551fbe1a03daa6f62bd0 not found: ID does not exist"
Oct 30 01:39:33.379571 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.379520    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7" podStartSLOduration=1.946600914 podStartE2EDuration="2.379506393s" totalImagesPullingTime="432.905479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:33.378025647 +0000 UTC m=+3485.716127572" watchObservedRunningTime="2026-10-30 01:39:33.379506393 +0000 UTC m=+3485.717608324"
Oct 30 01:39:33.414432 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.414400    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"]
Oct 30 01:39:33.420247 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.420181    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9da49f-openshift-default-556d9c56f5-lx45z"]
Oct 30 01:39:33.795289 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:33.795257    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aa394cb-eba6-4e15-9a0e-523e2379f7ac" path="/var/lib/kubelet/pods/3aa394cb-eba6-4e15-9a0e-523e2379f7ac/volumes"
Oct 30 01:39:40.409597 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:39:40.409562    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7"
Oct 30 01:39:49.044538 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:39:49.044501    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:40:03.359612 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:03.359567    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"]
Oct 30 01:40:03.360270 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:03.359834    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6" podUID="2fa04251-2153-46ef-b2fe-0b3583cfbaf9" containerName="istio-proxy" containerID="cri-o://a2a4908f4da3a6c0745571feb1b04badfe0f22bf9e22229756bffde82389e770" gracePeriod=30
Oct 30 01:40:08.648257 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.648230    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:40:08.658625 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.658592    4864 generic.go:356] "Generic (PLEG): container finished" podID="2fa04251-2153-46ef-b2fe-0b3583cfbaf9" containerID="a2a4908f4da3a6c0745571feb1b04badfe0f22bf9e22229756bffde82389e770" exitCode=0
Oct 30 01:40:08.658765 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.658638    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"
Oct 30 01:40:08.658765 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.658673    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6" event={"ID":"2fa04251-2153-46ef-b2fe-0b3583cfbaf9","Type":"ContainerDied","Data":"a2a4908f4da3a6c0745571feb1b04badfe0f22bf9e22229756bffde82389e770"}
Oct 30 01:40:08.658765 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.658742    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6" event={"ID":"2fa04251-2153-46ef-b2fe-0b3583cfbaf9","Type":"ContainerDied","Data":"9e8bc9f3f3202be144a543c13a7b807a6c4a44d0641d8883f3a6b0511d7460eb"}
Oct 30 01:40:08.658875 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.658811    4864 scope.go:122] "RemoveContainer" containerID="a2a4908f4da3a6c0745571feb1b04badfe0f22bf9e22229756bffde82389e770"
Oct 30 01:40:08.671167 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.671138    4864 scope.go:122] "RemoveContainer" containerID="a2a4908f4da3a6c0745571feb1b04badfe0f22bf9e22229756bffde82389e770"
Oct 30 01:40:08.671557 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:40:08.671525    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2a4908f4da3a6c0745571feb1b04badfe0f22bf9e22229756bffde82389e770\": container with ID starting with a2a4908f4da3a6c0745571feb1b04badfe0f22bf9e22229756bffde82389e770 not found: ID does not exist" containerID="a2a4908f4da3a6c0745571feb1b04badfe0f22bf9e22229756bffde82389e770"
Oct 30 01:40:08.671674 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.671569    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2a4908f4da3a6c0745571feb1b04badfe0f22bf9e22229756bffde82389e770"} err="failed to get container status \"a2a4908f4da3a6c0745571feb1b04badfe0f22bf9e22229756bffde82389e770\": rpc error: code = NotFound desc = could not find container \"a2a4908f4da3a6c0745571feb1b04badfe0f22bf9e22229756bffde82389e770\": container with ID starting with a2a4908f4da3a6c0745571feb1b04badfe0f22bf9e22229756bffde82389e770 not found: ID does not exist"
Oct 30 01:40:08.736819 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.736782    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-token\" (UniqueName: \"kubernetes.io/projected/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-token\") pod \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") "
Oct 30 01:40:08.737016 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.736879    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-workload-certs\") pod \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") "
Oct 30 01:40:08.737016 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.736973    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-data\") pod \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") "
Oct 30 01:40:08.737169 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.737111    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-podinfo\") pod \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") "
Oct 30 01:40:08.737169 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.737165    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istiod-ca-cert\") pod \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") "
Oct 30 01:40:08.737296 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.737214    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-credential-socket\") pod \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") "
Oct 30 01:40:08.737296 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.737266    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-kube-api-access-g5krk\" (UniqueName: \"kubernetes.io/projected/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-kube-api-access-g5krk\") pod \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") "
Oct 30 01:40:08.737296 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.737280    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-data" pod "2fa04251-2153-46ef-b2fe-0b3583cfbaf9" (UID: "2fa04251-2153-46ef-b2fe-0b3583cfbaf9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:40:08.737473 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.737302    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-workload-certs" pod "2fa04251-2153-46ef-b2fe-0b3583cfbaf9" (UID: "2fa04251-2153-46ef-b2fe-0b3583cfbaf9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:40:08.737533 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.737484    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-envoy\") pod \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") "
Oct 30 01:40:08.737533 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.737522    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-workload-socket\") pod \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\" (UID: \"2fa04251-2153-46ef-b2fe-0b3583cfbaf9\") "
Oct 30 01:40:08.737644 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.737561    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istiod-ca-cert" pod "2fa04251-2153-46ef-b2fe-0b3583cfbaf9" (UID: "2fa04251-2153-46ef-b2fe-0b3583cfbaf9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:40:08.737941 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.737912    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-workload-socket" pod "2fa04251-2153-46ef-b2fe-0b3583cfbaf9" (UID: "2fa04251-2153-46ef-b2fe-0b3583cfbaf9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:40:08.738383 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.738356    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:40:08.738490 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.738387    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:40:08.738490 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.738400    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:40:08.738490 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.738413    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:40:08.738671 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.738595    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-credential-socket" pod "2fa04251-2153-46ef-b2fe-0b3583cfbaf9" (UID: "2fa04251-2153-46ef-b2fe-0b3583cfbaf9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:40:08.740343 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.740316    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-podinfo" pod "2fa04251-2153-46ef-b2fe-0b3583cfbaf9" (UID: "2fa04251-2153-46ef-b2fe-0b3583cfbaf9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:40:08.740636 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.740613    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-kube-api-access-g5krk" pod "2fa04251-2153-46ef-b2fe-0b3583cfbaf9" (UID: "2fa04251-2153-46ef-b2fe-0b3583cfbaf9"). InnerVolumeSpecName "kube-api-access-g5krk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:40:08.740782 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.740623    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-envoy" pod "2fa04251-2153-46ef-b2fe-0b3583cfbaf9" (UID: "2fa04251-2153-46ef-b2fe-0b3583cfbaf9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:40:08.741851 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.741831    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-token" pod "2fa04251-2153-46ef-b2fe-0b3583cfbaf9" (UID: "2fa04251-2153-46ef-b2fe-0b3583cfbaf9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:40:08.839892 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.839860    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:40:08.839892 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.839889    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:40:08.839892 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.839899    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:40:08.840173 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.839907    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:40:08.840173 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:08.839919    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5krk\" (UniqueName: \"kubernetes.io/projected/2fa04251-2153-46ef-b2fe-0b3583cfbaf9-kube-api-access-g5krk\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:40:09.031679 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:09.031644    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"]
Oct 30 01:40:09.035719 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:09.035689    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-499482-openshift-default-6fff6f7b5-bfkl6"]
Oct 30 01:40:09.802169 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:40:09.802124    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fa04251-2153-46ef-b2fe-0b3583cfbaf9" path="/var/lib/kubelet/pods/2fa04251-2153-46ef-b2fe-0b3583cfbaf9/volumes"
Oct 30 01:40:19.040596 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:40:19.040555    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:40:49.067587 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:40:49.067547    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:41:19.040964 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:41:19.040923    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:41:28.554740 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:28.554712    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:41:28.568357 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:28.568334    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:41:31.498865 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:31.498833    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"]
Oct 30 01:41:31.499357 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:31.499034    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52" podUID="2c5848bd-c585-46df-93d6-d63b46d80c3b" containerName="istio-proxy" containerID="cri-o://bea4092a4088e53dcf2734de2f0dd0d8a5656f94f75e4b7fbef91d78e6b23c99" gracePeriod=30
Oct 30 01:41:34.414203 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:34.414149    4864 patch_prober.go:29] interesting pod/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:41:34.414582 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:34.414222    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52" podUID="2c5848bd-c585-46df-93d6-d63b46d80c3b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:41:36.784084 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:36.784000    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:41:36.985847 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:36.981251    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-token\" (UniqueName: \"kubernetes.io/projected/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-token\") pod \"2c5848bd-c585-46df-93d6-d63b46d80c3b\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") "
Oct 30 01:41:36.985847 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:36.981317    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-workload-certs\") pod \"2c5848bd-c585-46df-93d6-d63b46d80c3b\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") "
Oct 30 01:41:36.985847 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:36.981351    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-credential-socket\") pod \"2c5848bd-c585-46df-93d6-d63b46d80c3b\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") "
Oct 30 01:41:36.985847 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:36.981386    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-data\") pod \"2c5848bd-c585-46df-93d6-d63b46d80c3b\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") "
Oct 30 01:41:36.985847 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:36.981439    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-workload-socket\") pod \"2c5848bd-c585-46df-93d6-d63b46d80c3b\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") "
Oct 30 01:41:36.985847 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:36.981526    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c5848bd-c585-46df-93d6-d63b46d80c3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c5848bd-c585-46df-93d6-d63b46d80c3b-istiod-ca-cert\") pod \"2c5848bd-c585-46df-93d6-d63b46d80c3b\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") "
Oct 30 01:41:36.985847 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:36.981720    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-podinfo\") pod \"2c5848bd-c585-46df-93d6-d63b46d80c3b\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") "
Oct 30 01:41:36.985847 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:36.981788    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c5848bd-c585-46df-93d6-d63b46d80c3b-kube-api-access-dphvb\" (UniqueName: \"kubernetes.io/projected/2c5848bd-c585-46df-93d6-d63b46d80c3b-kube-api-access-dphvb\") pod \"2c5848bd-c585-46df-93d6-d63b46d80c3b\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") "
Oct 30 01:41:36.988967 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:36.981839    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-envoy\") pod \"2c5848bd-c585-46df-93d6-d63b46d80c3b\" (UID: \"2c5848bd-c585-46df-93d6-d63b46d80c3b\") "
Oct 30 01:41:36.988967 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:36.987348    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-data" pod "2c5848bd-c585-46df-93d6-d63b46d80c3b" (UID: "2c5848bd-c585-46df-93d6-d63b46d80c3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:36.988967 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:36.987887    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-workload-socket" pod "2c5848bd-c585-46df-93d6-d63b46d80c3b" (UID: "2c5848bd-c585-46df-93d6-d63b46d80c3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:36.988967 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:36.988196    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c5848bd-c585-46df-93d6-d63b46d80c3b-istiod-ca-cert" pod "2c5848bd-c585-46df-93d6-d63b46d80c3b" (UID: "2c5848bd-c585-46df-93d6-d63b46d80c3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:41:36.989325 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:36.988984    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-workload-certs" pod "2c5848bd-c585-46df-93d6-d63b46d80c3b" (UID: "2c5848bd-c585-46df-93d6-d63b46d80c3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:36.996950 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:36.994172    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-credential-socket" pod "2c5848bd-c585-46df-93d6-d63b46d80c3b" (UID: "2c5848bd-c585-46df-93d6-d63b46d80c3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:37.021678 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:37.021640    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c5848bd-c585-46df-93d6-d63b46d80c3b-kube-api-access-dphvb" pod "2c5848bd-c585-46df-93d6-d63b46d80c3b" (UID: "2c5848bd-c585-46df-93d6-d63b46d80c3b"). InnerVolumeSpecName "kube-api-access-dphvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:41:37.023547 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:37.023196    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-podinfo" pod "2c5848bd-c585-46df-93d6-d63b46d80c3b" (UID: "2c5848bd-c585-46df-93d6-d63b46d80c3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:41:37.023547 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:37.023203    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-envoy" pod "2c5848bd-c585-46df-93d6-d63b46d80c3b" (UID: "2c5848bd-c585-46df-93d6-d63b46d80c3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:37.034008 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:37.033973    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-token" pod "2c5848bd-c585-46df-93d6-d63b46d80c3b" (UID: "2c5848bd-c585-46df-93d6-d63b46d80c3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:41:37.088680 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:37.088645    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:41:37.088680 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:37.088675    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:41:37.088680 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:37.088684    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:41:37.089006 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:37.088694    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:41:37.089006 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:37.088708    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:41:37.089006 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:37.088720    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c5848bd-c585-46df-93d6-d63b46d80c3b-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:41:37.089006 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:37.088728    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:41:37.089006 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:37.088736    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dphvb\" (UniqueName: \"kubernetes.io/projected/2c5848bd-c585-46df-93d6-d63b46d80c3b-kube-api-access-dphvb\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:41:37.089006 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:37.088744    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c5848bd-c585-46df-93d6-d63b46d80c3b-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:41:37.140850 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:37.140815    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"]
Oct 30 01:41:37.368384 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:37.368289    4864 generic.go:356] "Generic (PLEG): container finished" podID="2c5848bd-c585-46df-93d6-d63b46d80c3b" containerID="bea4092a4088e53dcf2734de2f0dd0d8a5656f94f75e4b7fbef91d78e6b23c99" exitCode=0
Oct 30 01:41:37.368384 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:37.368355    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"
Oct 30 01:41:37.368384 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:37.368369    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52" event={"ID":"2c5848bd-c585-46df-93d6-d63b46d80c3b","Type":"ContainerDied","Data":"bea4092a4088e53dcf2734de2f0dd0d8a5656f94f75e4b7fbef91d78e6b23c99"}
Oct 30 01:41:37.368659 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:37.368424    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52" event={"ID":"2c5848bd-c585-46df-93d6-d63b46d80c3b","Type":"ContainerDied","Data":"a68158840a95b632f75fdbcde739dce2662646bcdb75b74f909d252898e4550e"}
Oct 30 01:41:37.368659 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:37.368447    4864 scope.go:122] "RemoveContainer" containerID="bea4092a4088e53dcf2734de2f0dd0d8a5656f94f75e4b7fbef91d78e6b23c99"
Oct 30 01:41:37.381840 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:37.381819    4864 scope.go:122] "RemoveContainer" containerID="bea4092a4088e53dcf2734de2f0dd0d8a5656f94f75e4b7fbef91d78e6b23c99"
Oct 30 01:41:37.382195 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:41:37.382172    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bea4092a4088e53dcf2734de2f0dd0d8a5656f94f75e4b7fbef91d78e6b23c99\": container with ID starting with bea4092a4088e53dcf2734de2f0dd0d8a5656f94f75e4b7fbef91d78e6b23c99 not found: ID does not exist" containerID="bea4092a4088e53dcf2734de2f0dd0d8a5656f94f75e4b7fbef91d78e6b23c99"
Oct 30 01:41:37.382291 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:37.382205    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea4092a4088e53dcf2734de2f0dd0d8a5656f94f75e4b7fbef91d78e6b23c99"} err="failed to get container status \"bea4092a4088e53dcf2734de2f0dd0d8a5656f94f75e4b7fbef91d78e6b23c99\": rpc error: code = NotFound desc = could not find container \"bea4092a4088e53dcf2734de2f0dd0d8a5656f94f75e4b7fbef91d78e6b23c99\": container with ID starting with bea4092a4088e53dcf2734de2f0dd0d8a5656f94f75e4b7fbef91d78e6b23c99 not found: ID does not exist"
Oct 30 01:41:37.423483 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:37.423440    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"]
Oct 30 01:41:37.433703 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:37.433673    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-df4db1-openshift-default-795cbf9c5f-2zh52"]
Oct 30 01:41:37.806138 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:37.806045    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c5848bd-c585-46df-93d6-d63b46d80c3b" path="/var/lib/kubelet/pods/2c5848bd-c585-46df-93d6-d63b46d80c3b/volumes"
Oct 30 01:41:44.595116 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:44.595060    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"]
Oct 30 01:41:44.595557 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:44.595357    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq" podUID="e6ae4638-c9e9-4d20-b699-feb3208662a2" containerName="istio-proxy" containerID="cri-o://b0aa74b32daead2adef84a913717fb332f4c7daf40684bcefd534bbb06aa77af" gracePeriod=30
Oct 30 01:41:49.042242 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:41:49.042198    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:41:50.167039 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.167010    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 01:41:50.304155 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.304038    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6ae4638-c9e9-4d20-b699-feb3208662a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ae4638-c9e9-4d20-b699-feb3208662a2-istiod-ca-cert\") pod \"e6ae4638-c9e9-4d20-b699-feb3208662a2\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") "
Oct 30 01:41:50.304155 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.304121    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-workload-certs\") pod \"e6ae4638-c9e9-4d20-b699-feb3208662a2\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") "
Oct 30 01:41:50.304155 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.304147    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-data\") pod \"e6ae4638-c9e9-4d20-b699-feb3208662a2\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") "
Oct 30 01:41:50.304458 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.304255    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-credential-socket\") pod \"e6ae4638-c9e9-4d20-b699-feb3208662a2\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") "
Oct 30 01:41:50.304458 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.304291    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-token\" (UniqueName: \"kubernetes.io/projected/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-token\") pod \"e6ae4638-c9e9-4d20-b699-feb3208662a2\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") "
Oct 30 01:41:50.304458 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.304316    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-workload-socket\") pod \"e6ae4638-c9e9-4d20-b699-feb3208662a2\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") "
Oct 30 01:41:50.304458 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.304399    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-podinfo\") pod \"e6ae4638-c9e9-4d20-b699-feb3208662a2\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") "
Oct 30 01:41:50.304458 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.304368    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-workload-certs" pod "e6ae4638-c9e9-4d20-b699-feb3208662a2" (UID: "e6ae4638-c9e9-4d20-b699-feb3208662a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:50.304458 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.304455    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-envoy\") pod \"e6ae4638-c9e9-4d20-b699-feb3208662a2\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") "
Oct 30 01:41:50.304761 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.304494    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-data" pod "e6ae4638-c9e9-4d20-b699-feb3208662a2" (UID: "e6ae4638-c9e9-4d20-b699-feb3208662a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:50.304761 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.304572    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-credential-socket" pod "e6ae4638-c9e9-4d20-b699-feb3208662a2" (UID: "e6ae4638-c9e9-4d20-b699-feb3208662a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:50.304761 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.304628    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-workload-socket" pod "e6ae4638-c9e9-4d20-b699-feb3208662a2" (UID: "e6ae4638-c9e9-4d20-b699-feb3208662a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:50.304761 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.304635    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ae4638-c9e9-4d20-b699-feb3208662a2-istiod-ca-cert" pod "e6ae4638-c9e9-4d20-b699-feb3208662a2" (UID: "e6ae4638-c9e9-4d20-b699-feb3208662a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:41:50.304761 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.304666    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ae4638-c9e9-4d20-b699-feb3208662a2-kube-api-access-g998b\" (UniqueName: \"kubernetes.io/projected/e6ae4638-c9e9-4d20-b699-feb3208662a2-kube-api-access-g998b\") pod \"e6ae4638-c9e9-4d20-b699-feb3208662a2\" (UID: \"e6ae4638-c9e9-4d20-b699-feb3208662a2\") "
Oct 30 01:41:50.305378 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.305358    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ae4638-c9e9-4d20-b699-feb3208662a2-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:41:50.305378 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.305377    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:41:50.305550 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.305387    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:41:50.305550 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.305398    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:41:50.305550 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.305406    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:41:50.306611 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.306584    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-token" pod "e6ae4638-c9e9-4d20-b699-feb3208662a2" (UID: "e6ae4638-c9e9-4d20-b699-feb3208662a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:41:50.306693 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.306650    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-envoy" pod "e6ae4638-c9e9-4d20-b699-feb3208662a2" (UID: "e6ae4638-c9e9-4d20-b699-feb3208662a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:50.307066 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.307041    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-podinfo" pod "e6ae4638-c9e9-4d20-b699-feb3208662a2" (UID: "e6ae4638-c9e9-4d20-b699-feb3208662a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:41:50.307179 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.307067    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ae4638-c9e9-4d20-b699-feb3208662a2-kube-api-access-g998b" pod "e6ae4638-c9e9-4d20-b699-feb3208662a2" (UID: "e6ae4638-c9e9-4d20-b699-feb3208662a2"). InnerVolumeSpecName "kube-api-access-g998b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:41:50.406667 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.406637    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:41:50.406667 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.406665    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:41:50.406667 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.406674    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ae4638-c9e9-4d20-b699-feb3208662a2-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:41:50.406919 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.406683    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g998b\" (UniqueName: \"kubernetes.io/projected/e6ae4638-c9e9-4d20-b699-feb3208662a2-kube-api-access-g998b\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:41:50.505533 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.505483    4864 generic.go:356] "Generic (PLEG): container finished" podID="e6ae4638-c9e9-4d20-b699-feb3208662a2" containerID="b0aa74b32daead2adef84a913717fb332f4c7daf40684bcefd534bbb06aa77af" exitCode=0
Oct 30 01:41:50.505698 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.505582    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"
Oct 30 01:41:50.505698 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.505616    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq" event={"ID":"e6ae4638-c9e9-4d20-b699-feb3208662a2","Type":"ContainerDied","Data":"b0aa74b32daead2adef84a913717fb332f4c7daf40684bcefd534bbb06aa77af"}
Oct 30 01:41:50.505698 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.505668    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq" event={"ID":"e6ae4638-c9e9-4d20-b699-feb3208662a2","Type":"ContainerDied","Data":"b0b106acdd9445a13f75c0a44c656803e6d3644b36c12e5956b968634b692447"}
Oct 30 01:41:50.505698 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.505689    4864 scope.go:122] "RemoveContainer" containerID="b0aa74b32daead2adef84a913717fb332f4c7daf40684bcefd534bbb06aa77af"
Oct 30 01:41:50.527841 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.527817    4864 scope.go:122] "RemoveContainer" containerID="44b2431d0824a1af95eab7cedd2fc0bd4c7690e383cf9c6338cb0e131aca4102"
Oct 30 01:41:50.539210 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.538928    4864 scope.go:122] "RemoveContainer" containerID="b0aa74b32daead2adef84a913717fb332f4c7daf40684bcefd534bbb06aa77af"
Oct 30 01:41:50.539343 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:41:50.539320    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0aa74b32daead2adef84a913717fb332f4c7daf40684bcefd534bbb06aa77af\": container with ID starting with b0aa74b32daead2adef84a913717fb332f4c7daf40684bcefd534bbb06aa77af not found: ID does not exist" containerID="b0aa74b32daead2adef84a913717fb332f4c7daf40684bcefd534bbb06aa77af"
Oct 30 01:41:50.539390 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.539355    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0aa74b32daead2adef84a913717fb332f4c7daf40684bcefd534bbb06aa77af"} err="failed to get container status \"b0aa74b32daead2adef84a913717fb332f4c7daf40684bcefd534bbb06aa77af\": rpc error: code = NotFound desc = could not find container \"b0aa74b32daead2adef84a913717fb332f4c7daf40684bcefd534bbb06aa77af\": container with ID starting with b0aa74b32daead2adef84a913717fb332f4c7daf40684bcefd534bbb06aa77af not found: ID does not exist"
Oct 30 01:41:50.539390 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.539376    4864 scope.go:122] "RemoveContainer" containerID="44b2431d0824a1af95eab7cedd2fc0bd4c7690e383cf9c6338cb0e131aca4102"
Oct 30 01:41:50.539633 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:41:50.539619    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44b2431d0824a1af95eab7cedd2fc0bd4c7690e383cf9c6338cb0e131aca4102\": container with ID starting with 44b2431d0824a1af95eab7cedd2fc0bd4c7690e383cf9c6338cb0e131aca4102 not found: ID does not exist" containerID="44b2431d0824a1af95eab7cedd2fc0bd4c7690e383cf9c6338cb0e131aca4102"
Oct 30 01:41:50.539676 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.539637    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44b2431d0824a1af95eab7cedd2fc0bd4c7690e383cf9c6338cb0e131aca4102"} err="failed to get container status \"44b2431d0824a1af95eab7cedd2fc0bd4c7690e383cf9c6338cb0e131aca4102\": rpc error: code = NotFound desc = could not find container \"44b2431d0824a1af95eab7cedd2fc0bd4c7690e383cf9c6338cb0e131aca4102\": container with ID starting with 44b2431d0824a1af95eab7cedd2fc0bd4c7690e383cf9c6338cb0e131aca4102 not found: ID does not exist"
Oct 30 01:41:50.563134 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.562797    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"]
Oct 30 01:41:50.567196 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:50.567166    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-q4qlq"]
Oct 30 01:41:51.822146 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:51.821878    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6ae4638-c9e9-4d20-b699-feb3208662a2" path="/var/lib/kubelet/pods/e6ae4638-c9e9-4d20-b699-feb3208662a2/volumes"
Oct 30 01:41:52.162435 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.162338    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9" podUID="916074d0-6586-417a-baf6-df529fdd58b8" containerName="main" containerID="cri-o://e3538048cdbc8fd90adf9b4c57df2858210a5727b63ea545bee0282d9b172671" gracePeriod=45
Oct 30 01:41:52.331092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.331039    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9" podUID="916074d0-6586-417a-baf6-df529fdd58b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.158:8000/ready\": dial tcp 10.132.0.158:8000: connect: connection refused"
Oct 30 01:41:52.462760 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.462734    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:41:52.526381 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.526348    4864 generic.go:356] "Generic (PLEG): container finished" podID="916074d0-6586-417a-baf6-df529fdd58b8" containerID="e3538048cdbc8fd90adf9b4c57df2858210a5727b63ea545bee0282d9b172671" exitCode=0
Oct 30 01:41:52.526567 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.526400    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"
Oct 30 01:41:52.526567 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.526431    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9" event={"ID":"916074d0-6586-417a-baf6-df529fdd58b8","Type":"ContainerDied","Data":"e3538048cdbc8fd90adf9b4c57df2858210a5727b63ea545bee0282d9b172671"}
Oct 30 01:41:52.526567 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.526474    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9" event={"ID":"916074d0-6586-417a-baf6-df529fdd58b8","Type":"ContainerDied","Data":"b57d88f2593d19490d0759cff10f72eaa388765f9b0366b77d7ff4e0639e5365"}
Oct 30 01:41:52.526567 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.526496    4864 scope.go:122] "RemoveContainer" containerID="e3538048cdbc8fd90adf9b4c57df2858210a5727b63ea545bee0282d9b172671"
Oct 30 01:41:52.536875 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.536855    4864 scope.go:122] "RemoveContainer" containerID="e3538048cdbc8fd90adf9b4c57df2858210a5727b63ea545bee0282d9b172671"
Oct 30 01:41:52.537211 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:41:52.537181    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3538048cdbc8fd90adf9b4c57df2858210a5727b63ea545bee0282d9b172671\": container with ID starting with e3538048cdbc8fd90adf9b4c57df2858210a5727b63ea545bee0282d9b172671 not found: ID does not exist" containerID="e3538048cdbc8fd90adf9b4c57df2858210a5727b63ea545bee0282d9b172671"
Oct 30 01:41:52.537300 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.537223    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3538048cdbc8fd90adf9b4c57df2858210a5727b63ea545bee0282d9b172671"} err="failed to get container status \"e3538048cdbc8fd90adf9b4c57df2858210a5727b63ea545bee0282d9b172671\": rpc error: code = NotFound desc = could not find container \"e3538048cdbc8fd90adf9b4c57df2858210a5727b63ea545bee0282d9b172671\": container with ID starting with e3538048cdbc8fd90adf9b4c57df2858210a5727b63ea545bee0282d9b172671 not found: ID does not exist"
Oct 30 01:41:52.543862 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.543840    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-tmp-dir\") pod \"916074d0-6586-417a-baf6-df529fdd58b8\" (UID: \"916074d0-6586-417a-baf6-df529fdd58b8\") "
Oct 30 01:41:52.543968 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.543874    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-home\" (UniqueName: \"kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-home\") pod \"916074d0-6586-417a-baf6-df529fdd58b8\" (UID: \"916074d0-6586-417a-baf6-df529fdd58b8\") "
Oct 30 01:41:52.543968 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.543939    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916074d0-6586-417a-baf6-df529fdd58b8-kube-api-access-547f6\" (UniqueName: \"kubernetes.io/projected/916074d0-6586-417a-baf6-df529fdd58b8-kube-api-access-547f6\") pod \"916074d0-6586-417a-baf6-df529fdd58b8\" (UID: \"916074d0-6586-417a-baf6-df529fdd58b8\") "
Oct 30 01:41:52.544104 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.544002    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-dshm\") pod \"916074d0-6586-417a-baf6-df529fdd58b8\" (UID: \"916074d0-6586-417a-baf6-df529fdd58b8\") "
Oct 30 01:41:52.544104 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.544099    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/916074d0-6586-417a-baf6-df529fdd58b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/916074d0-6586-417a-baf6-df529fdd58b8-tls-certs\") pod \"916074d0-6586-417a-baf6-df529fdd58b8\" (UID: \"916074d0-6586-417a-baf6-df529fdd58b8\") "
Oct 30 01:41:52.544223 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.544155    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-tmp-dir" pod "916074d0-6586-417a-baf6-df529fdd58b8" (UID: "916074d0-6586-417a-baf6-df529fdd58b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:52.544223 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.544171    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-home" pod "916074d0-6586-417a-baf6-df529fdd58b8" (UID: "916074d0-6586-417a-baf6-df529fdd58b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:52.544307 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.544287    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-model-cache\") pod \"916074d0-6586-417a-baf6-df529fdd58b8\" (UID: \"916074d0-6586-417a-baf6-df529fdd58b8\") "
Oct 30 01:41:52.544627 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.544605    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-model-cache" pod "916074d0-6586-417a-baf6-df529fdd58b8" (UID: "916074d0-6586-417a-baf6-df529fdd58b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:52.545026 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.545006    4864 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-model-cache\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:41:52.545159 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.545030    4864 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-tmp-dir\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:41:52.545159 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.545043    4864 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-home\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:41:52.546827 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.546801    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-dshm" pod "916074d0-6586-417a-baf6-df529fdd58b8" (UID: "916074d0-6586-417a-baf6-df529fdd58b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:52.546912 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.546893    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/916074d0-6586-417a-baf6-df529fdd58b8-tls-certs" pod "916074d0-6586-417a-baf6-df529fdd58b8" (UID: "916074d0-6586-417a-baf6-df529fdd58b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:41:52.547001 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.546984    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916074d0-6586-417a-baf6-df529fdd58b8-kube-api-access-547f6" pod "916074d0-6586-417a-baf6-df529fdd58b8" (UID: "916074d0-6586-417a-baf6-df529fdd58b8"). InnerVolumeSpecName "kube-api-access-547f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:41:52.646767 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.646730    4864 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/916074d0-6586-417a-baf6-df529fdd58b8-tls-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:41:52.646767 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.646765    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-547f6\" (UniqueName: \"kubernetes.io/projected/916074d0-6586-417a-baf6-df529fdd58b8-kube-api-access-547f6\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:41:52.646963 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.646778    4864 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/916074d0-6586-417a-baf6-df529fdd58b8-dshm\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:41:52.869827 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.869790    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"]
Oct 30 01:41:52.880068 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:52.880036    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-415c6b/test-model-415c6b-kserve-85cd844477-lwrv9"]
Oct 30 01:41:53.778339 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:53.778301    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"]
Oct 30 01:41:53.802285 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:53.802252    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="916074d0-6586-417a-baf6-df529fdd58b8" path="/var/lib/kubelet/pods/916074d0-6586-417a-baf6-df529fdd58b8/volumes"
Oct 30 01:41:58.407468 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:58.407427    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"]
Oct 30 01:41:58.407967 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:58.407672    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7" podUID="db3e7716-143c-4e34-8d22-daae99a75109" containerName="istio-proxy" containerID="cri-o://61b2c1602aafc4148bb8e91e1e5c61a39565d65348270b039f3a742e168835b6" gracePeriod=30
Oct 30 01:41:59.431599 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:59.431558    4864 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:41:59.432005 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:41:59.431617    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7" podUID="db3e7716-143c-4e34-8d22-daae99a75109" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:42:03.613032 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.612971    4864 generic.go:356] "Generic (PLEG): container finished" podID="db3e7716-143c-4e34-8d22-daae99a75109" containerID="61b2c1602aafc4148bb8e91e1e5c61a39565d65348270b039f3a742e168835b6" exitCode=0
Oct 30 01:42:03.613521 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.613122    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7" event={"ID":"db3e7716-143c-4e34-8d22-daae99a75109","Type":"ContainerDied","Data":"61b2c1602aafc4148bb8e91e1e5c61a39565d65348270b039f3a742e168835b6"}
Oct 30 01:42:03.695801 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.695742    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 01:42:03.836408 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.836376    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-workload-socket\") pod \"db3e7716-143c-4e34-8d22-daae99a75109\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") "
Oct 30 01:42:03.836582 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.836470    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db3e7716-143c-4e34-8d22-daae99a75109-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db3e7716-143c-4e34-8d22-daae99a75109-istio-podinfo\") pod \"db3e7716-143c-4e34-8d22-daae99a75109\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") "
Oct 30 01:42:03.836582 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.836503    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-credential-socket\") pod \"db3e7716-143c-4e34-8d22-daae99a75109\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") "
Oct 30 01:42:03.836582 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.836549    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-workload-certs\") pod \"db3e7716-143c-4e34-8d22-daae99a75109\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") "
Oct 30 01:42:03.836730 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.836588    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db3e7716-143c-4e34-8d22-daae99a75109-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db3e7716-143c-4e34-8d22-daae99a75109-istiod-ca-cert\") pod \"db3e7716-143c-4e34-8d22-daae99a75109\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") "
Oct 30 01:42:03.836730 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.836636    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-istio-data\") pod \"db3e7716-143c-4e34-8d22-daae99a75109\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") "
Oct 30 01:42:03.836730 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.836657    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-istio-envoy\") pod \"db3e7716-143c-4e34-8d22-daae99a75109\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") "
Oct 30 01:42:03.836730 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.836674    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-workload-socket" pod "db3e7716-143c-4e34-8d22-daae99a75109" (UID: "db3e7716-143c-4e34-8d22-daae99a75109"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:42:03.836730 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.836688    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db3e7716-143c-4e34-8d22-daae99a75109-kube-api-access-rvwpl\" (UniqueName: \"kubernetes.io/projected/db3e7716-143c-4e34-8d22-daae99a75109-kube-api-access-rvwpl\") pod \"db3e7716-143c-4e34-8d22-daae99a75109\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") "
Oct 30 01:42:03.836987 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.836802    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-credential-socket" pod "db3e7716-143c-4e34-8d22-daae99a75109" (UID: "db3e7716-143c-4e34-8d22-daae99a75109"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:42:03.836987 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.836845    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-workload-certs" pod "db3e7716-143c-4e34-8d22-daae99a75109" (UID: "db3e7716-143c-4e34-8d22-daae99a75109"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:42:03.836987 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.836880    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db3e7716-143c-4e34-8d22-daae99a75109-istio-token\" (UniqueName: \"kubernetes.io/projected/db3e7716-143c-4e34-8d22-daae99a75109-istio-token\") pod \"db3e7716-143c-4e34-8d22-daae99a75109\" (UID: \"db3e7716-143c-4e34-8d22-daae99a75109\") "
Oct 30 01:42:03.837176 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.837105    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db3e7716-143c-4e34-8d22-daae99a75109-istiod-ca-cert" pod "db3e7716-143c-4e34-8d22-daae99a75109" (UID: "db3e7716-143c-4e34-8d22-daae99a75109"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:42:03.837242 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.837176    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-istio-data" pod "db3e7716-143c-4e34-8d22-daae99a75109" (UID: "db3e7716-143c-4e34-8d22-daae99a75109"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:42:03.837680 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.837651    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:03.837820 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.837685    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:03.837820 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.837703    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db3e7716-143c-4e34-8d22-daae99a75109-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:03.837820 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.837717    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:03.837820 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.837731    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:03.838987 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.838961    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db3e7716-143c-4e34-8d22-daae99a75109-istio-podinfo" pod "db3e7716-143c-4e34-8d22-daae99a75109" (UID: "db3e7716-143c-4e34-8d22-daae99a75109"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:42:03.839090 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.838982    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3e7716-143c-4e34-8d22-daae99a75109-kube-api-access-rvwpl" pod "db3e7716-143c-4e34-8d22-daae99a75109" (UID: "db3e7716-143c-4e34-8d22-daae99a75109"). InnerVolumeSpecName "kube-api-access-rvwpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:42:03.839160 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.839094    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3e7716-143c-4e34-8d22-daae99a75109-istio-token" pod "db3e7716-143c-4e34-8d22-daae99a75109" (UID: "db3e7716-143c-4e34-8d22-daae99a75109"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:42:03.839160 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.839124    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-istio-envoy" pod "db3e7716-143c-4e34-8d22-daae99a75109" (UID: "db3e7716-143c-4e34-8d22-daae99a75109"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:42:03.939409 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.939373    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db3e7716-143c-4e34-8d22-daae99a75109-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:03.939409 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.939411    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db3e7716-143c-4e34-8d22-daae99a75109-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:03.939732 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.939426    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvwpl\" (UniqueName: \"kubernetes.io/projected/db3e7716-143c-4e34-8d22-daae99a75109-kube-api-access-rvwpl\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:03.939732 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:03.939441    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3e7716-143c-4e34-8d22-daae99a75109-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:04.622601 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:04.622572    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"
Oct 30 01:42:04.623067 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:04.622602    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7" event={"ID":"db3e7716-143c-4e34-8d22-daae99a75109","Type":"ContainerDied","Data":"301123f0fab905562bc6f817eaca645887fa4f77fee227ab08813ff7729c0650"}
Oct 30 01:42:04.623067 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:04.622678    4864 scope.go:122] "RemoveContainer" containerID="61b2c1602aafc4148bb8e91e1e5c61a39565d65348270b039f3a742e168835b6"
Oct 30 01:42:04.683873 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:04.683838    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"]
Oct 30 01:42:04.689874 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:04.689843    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rhrp7"]
Oct 30 01:42:05.794920 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:05.794888    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db3e7716-143c-4e34-8d22-daae99a75109" path="/var/lib/kubelet/pods/db3e7716-143c-4e34-8d22-daae99a75109/volumes"
Oct 30 01:42:06.938158 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:06.938126    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"]
Oct 30 01:42:06.938649 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:06.938367    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn" podUID="0d79aa0b-3be0-4837-8182-e1e940a4d73a" containerName="istio-proxy" containerID="cri-o://3b76fcda9bc226478d5db559ad4e6233435ea901203dc71040112a0c7b5d29bf" gracePeriod=30
Oct 30 01:42:08.809365 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:08.809327    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42" podUID="056d952a-9dde-494f-9504-7e104751184b" containerName="main" containerID="cri-o://646311bbdbc3ecd1ce50ed0088d43338e0933bebddb63e28d3d915104009f6a3" gracePeriod=45
Oct 30 01:42:09.004337 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.004295    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42" podUID="056d952a-9dde-494f-9504-7e104751184b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": dial tcp 10.132.0.160:8000: connect: connection refused"
Oct 30 01:42:09.137647 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.137615    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:42:09.227990 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.227951    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/056d952a-9dde-494f-9504-7e104751184b-tls-certs\" (UniqueName: \"kubernetes.io/secret/056d952a-9dde-494f-9504-7e104751184b-tls-certs\") pod \"056d952a-9dde-494f-9504-7e104751184b\" (UID: \"056d952a-9dde-494f-9504-7e104751184b\") "
Oct 30 01:42:09.228219 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.228054    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-dshm\") pod \"056d952a-9dde-494f-9504-7e104751184b\" (UID: \"056d952a-9dde-494f-9504-7e104751184b\") "
Oct 30 01:42:09.228676 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.228642    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-model-cache\") pod \"056d952a-9dde-494f-9504-7e104751184b\" (UID: \"056d952a-9dde-494f-9504-7e104751184b\") "
Oct 30 01:42:09.229250 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.229229    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-home\" (UniqueName: \"kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-home\") pod \"056d952a-9dde-494f-9504-7e104751184b\" (UID: \"056d952a-9dde-494f-9504-7e104751184b\") "
Oct 30 01:42:09.229543 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.229135    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-model-cache" pod "056d952a-9dde-494f-9504-7e104751184b" (UID: "056d952a-9dde-494f-9504-7e104751184b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:42:09.229889 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.229852    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-home" pod "056d952a-9dde-494f-9504-7e104751184b" (UID: "056d952a-9dde-494f-9504-7e104751184b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:42:09.230043 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.230024    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/056d952a-9dde-494f-9504-7e104751184b-kube-api-access-rn8hq\" (UniqueName: \"kubernetes.io/projected/056d952a-9dde-494f-9504-7e104751184b-kube-api-access-rn8hq\") pod \"056d952a-9dde-494f-9504-7e104751184b\" (UID: \"056d952a-9dde-494f-9504-7e104751184b\") "
Oct 30 01:42:09.230214 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.230189    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-tmp-dir\") pod \"056d952a-9dde-494f-9504-7e104751184b\" (UID: \"056d952a-9dde-494f-9504-7e104751184b\") "
Oct 30 01:42:09.231023 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.230997    4864 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-home\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:09.231158 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.231030    4864 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-model-cache\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:09.231278 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.231257    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-tmp-dir" pod "056d952a-9dde-494f-9504-7e104751184b" (UID: "056d952a-9dde-494f-9504-7e104751184b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:42:09.233701 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.233315    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056d952a-9dde-494f-9504-7e104751184b-tls-certs" pod "056d952a-9dde-494f-9504-7e104751184b" (UID: "056d952a-9dde-494f-9504-7e104751184b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:42:09.233701 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.233575    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-dshm" pod "056d952a-9dde-494f-9504-7e104751184b" (UID: "056d952a-9dde-494f-9504-7e104751184b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:42:09.233701 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.233678    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056d952a-9dde-494f-9504-7e104751184b-kube-api-access-rn8hq" pod "056d952a-9dde-494f-9504-7e104751184b" (UID: "056d952a-9dde-494f-9504-7e104751184b"). InnerVolumeSpecName "kube-api-access-rn8hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:42:09.332590 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.332552    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn8hq\" (UniqueName: \"kubernetes.io/projected/056d952a-9dde-494f-9504-7e104751184b-kube-api-access-rn8hq\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:09.332590 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.332585    4864 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-tmp-dir\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:09.332590 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.332602    4864 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/056d952a-9dde-494f-9504-7e104751184b-tls-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:09.332879 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.332611    4864 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/056d952a-9dde-494f-9504-7e104751184b-dshm\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:09.660416 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.660378    4864 generic.go:356] "Generic (PLEG): container finished" podID="056d952a-9dde-494f-9504-7e104751184b" containerID="646311bbdbc3ecd1ce50ed0088d43338e0933bebddb63e28d3d915104009f6a3" exitCode=0
Oct 30 01:42:09.660603 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.660431    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"
Oct 30 01:42:09.660603 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.660468    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42" event={"ID":"056d952a-9dde-494f-9504-7e104751184b","Type":"ContainerDied","Data":"646311bbdbc3ecd1ce50ed0088d43338e0933bebddb63e28d3d915104009f6a3"}
Oct 30 01:42:09.660603 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.660516    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42" event={"ID":"056d952a-9dde-494f-9504-7e104751184b","Type":"ContainerDied","Data":"5ca026abfe3bb10048537147f333def3f72aed98d96b1e71cf16ec68c826b911"}
Oct 30 01:42:09.660603 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.660544    4864 scope.go:122] "RemoveContainer" containerID="646311bbdbc3ecd1ce50ed0088d43338e0933bebddb63e28d3d915104009f6a3"
Oct 30 01:42:09.671788 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.671766    4864 scope.go:122] "RemoveContainer" containerID="646311bbdbc3ecd1ce50ed0088d43338e0933bebddb63e28d3d915104009f6a3"
Oct 30 01:42:09.672050 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:42:09.672027    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"646311bbdbc3ecd1ce50ed0088d43338e0933bebddb63e28d3d915104009f6a3\": container with ID starting with 646311bbdbc3ecd1ce50ed0088d43338e0933bebddb63e28d3d915104009f6a3 not found: ID does not exist" containerID="646311bbdbc3ecd1ce50ed0088d43338e0933bebddb63e28d3d915104009f6a3"
Oct 30 01:42:09.672142 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.672061    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646311bbdbc3ecd1ce50ed0088d43338e0933bebddb63e28d3d915104009f6a3"} err="failed to get container status \"646311bbdbc3ecd1ce50ed0088d43338e0933bebddb63e28d3d915104009f6a3\": rpc error: code = NotFound desc = could not find container \"646311bbdbc3ecd1ce50ed0088d43338e0933bebddb63e28d3d915104009f6a3\": container with ID starting with 646311bbdbc3ecd1ce50ed0088d43338e0933bebddb63e28d3d915104009f6a3 not found: ID does not exist"
Oct 30 01:42:09.707591 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.707562    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"]
Oct 30 01:42:09.711134 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.711111    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ab0a2c/test-model-ab0a2c-kserve-85cbb78c6f-66j42"]
Oct 30 01:42:09.795581 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.795542    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="056d952a-9dde-494f-9504-7e104751184b" path="/var/lib/kubelet/pods/056d952a-9dde-494f-9504-7e104751184b/volumes"
Oct 30 01:42:09.964038 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.963997    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"]
Oct 30 01:42:09.964550 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:09.964282    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf" podUID="6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" containerName="istio-proxy" containerID="cri-o://141e1d76ee6cf0f0238cf91bbe5fed92c12e621a7ea474ad428dc6f2cdbff24f" gracePeriod=30
Oct 30 01:42:12.213161 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.213093    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:42:12.288090 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.288041    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-token\" (UniqueName: \"kubernetes.io/projected/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-token\") pod \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") "
Oct 30 01:42:12.288269 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.288145    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-workload-certs\") pod \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") "
Oct 30 01:42:12.288269 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.288177    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-podinfo\") pod \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") "
Oct 30 01:42:12.288269 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.288251    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-envoy\") pod \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") "
Oct 30 01:42:12.288444 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.288322    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-credential-socket\") pod \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") "
Oct 30 01:42:12.288444 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.288378    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d79aa0b-3be0-4837-8182-e1e940a4d73a-kube-api-access-dwl52\" (UniqueName: \"kubernetes.io/projected/0d79aa0b-3be0-4837-8182-e1e940a4d73a-kube-api-access-dwl52\") pod \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") "
Oct 30 01:42:12.288545 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.288463    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-workload-socket\") pod \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") "
Oct 30 01:42:12.288545 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.288493    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-workload-certs" pod "0d79aa0b-3be0-4837-8182-e1e940a4d73a" (UID: "0d79aa0b-3be0-4837-8182-e1e940a4d73a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:42:12.288545 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.288512    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-data\") pod \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") "
Oct 30 01:42:12.288716 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.288575    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istiod-ca-cert\") pod \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\" (UID: \"0d79aa0b-3be0-4837-8182-e1e940a4d73a\") "
Oct 30 01:42:12.288774 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.288713    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-credential-socket" pod "0d79aa0b-3be0-4837-8182-e1e940a4d73a" (UID: "0d79aa0b-3be0-4837-8182-e1e940a4d73a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:42:12.289392 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.288919    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-workload-socket" pod "0d79aa0b-3be0-4837-8182-e1e940a4d73a" (UID: "0d79aa0b-3be0-4837-8182-e1e940a4d73a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:42:12.289392 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.289360    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-data" pod "0d79aa0b-3be0-4837-8182-e1e940a4d73a" (UID: "0d79aa0b-3be0-4837-8182-e1e940a4d73a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:42:12.289392 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.289369    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istiod-ca-cert" pod "0d79aa0b-3be0-4837-8182-e1e940a4d73a" (UID: "0d79aa0b-3be0-4837-8182-e1e940a4d73a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:42:12.289656 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.289517    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:12.289656 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.289538    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:12.289656 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.289554    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:12.289656 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.289568    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:12.289656 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.289582    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:12.292310 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.292261    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-podinfo" pod "0d79aa0b-3be0-4837-8182-e1e940a4d73a" (UID: "0d79aa0b-3be0-4837-8182-e1e940a4d73a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:42:12.292410 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.292267    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-token" pod "0d79aa0b-3be0-4837-8182-e1e940a4d73a" (UID: "0d79aa0b-3be0-4837-8182-e1e940a4d73a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:42:12.293268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.293189    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-envoy" pod "0d79aa0b-3be0-4837-8182-e1e940a4d73a" (UID: "0d79aa0b-3be0-4837-8182-e1e940a4d73a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:42:12.293268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.293232    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d79aa0b-3be0-4837-8182-e1e940a4d73a-kube-api-access-dwl52" pod "0d79aa0b-3be0-4837-8182-e1e940a4d73a" (UID: "0d79aa0b-3be0-4837-8182-e1e940a4d73a"). InnerVolumeSpecName "kube-api-access-dwl52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:42:12.391225 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.391190    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:12.391225 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.391220    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:12.391225 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.391229    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d79aa0b-3be0-4837-8182-e1e940a4d73a-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:12.391458 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.391238    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwl52\" (UniqueName: \"kubernetes.io/projected/0d79aa0b-3be0-4837-8182-e1e940a4d73a-kube-api-access-dwl52\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:12.684692 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.684601    4864 generic.go:356] "Generic (PLEG): container finished" podID="0d79aa0b-3be0-4837-8182-e1e940a4d73a" containerID="3b76fcda9bc226478d5db559ad4e6233435ea901203dc71040112a0c7b5d29bf" exitCode=0
Oct 30 01:42:12.684692 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.684647    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"
Oct 30 01:42:12.684692 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.684665    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn" event={"ID":"0d79aa0b-3be0-4837-8182-e1e940a4d73a","Type":"ContainerDied","Data":"3b76fcda9bc226478d5db559ad4e6233435ea901203dc71040112a0c7b5d29bf"}
Oct 30 01:42:12.684912 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.684712    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn" event={"ID":"0d79aa0b-3be0-4837-8182-e1e940a4d73a","Type":"ContainerDied","Data":"5e5adca015c5985235e9c10839fb791ff391c72c10ba51f5a28ea8af0ef6742b"}
Oct 30 01:42:12.684912 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.684739    4864 scope.go:122] "RemoveContainer" containerID="3b76fcda9bc226478d5db559ad4e6233435ea901203dc71040112a0c7b5d29bf"
Oct 30 01:42:12.697617 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.697591    4864 scope.go:122] "RemoveContainer" containerID="3b76fcda9bc226478d5db559ad4e6233435ea901203dc71040112a0c7b5d29bf"
Oct 30 01:42:12.697945 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:42:12.697924    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b76fcda9bc226478d5db559ad4e6233435ea901203dc71040112a0c7b5d29bf\": container with ID starting with 3b76fcda9bc226478d5db559ad4e6233435ea901203dc71040112a0c7b5d29bf not found: ID does not exist" containerID="3b76fcda9bc226478d5db559ad4e6233435ea901203dc71040112a0c7b5d29bf"
Oct 30 01:42:12.697995 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.697954    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b76fcda9bc226478d5db559ad4e6233435ea901203dc71040112a0c7b5d29bf"} err="failed to get container status \"3b76fcda9bc226478d5db559ad4e6233435ea901203dc71040112a0c7b5d29bf\": rpc error: code = NotFound desc = could not find container \"3b76fcda9bc226478d5db559ad4e6233435ea901203dc71040112a0c7b5d29bf\": container with ID starting with 3b76fcda9bc226478d5db559ad4e6233435ea901203dc71040112a0c7b5d29bf not found: ID does not exist"
Oct 30 01:42:12.733633 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.733602    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"]
Oct 30 01:42:12.743714 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:12.743683    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zz9kn"]
Oct 30 01:42:13.644836 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:13.644793    4864 patch_prober.go:29] interesting pod/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:42:13.645237 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:13.644858    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf" podUID="6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:42:13.795025 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:13.794990    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d79aa0b-3be0-4837-8182-e1e940a4d73a" path="/var/lib/kubelet/pods/0d79aa0b-3be0-4837-8182-e1e940a4d73a/volumes"
Oct 30 01:42:15.245272 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.245204    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:42:15.337932 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.337888    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-credential-socket\") pod \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") "
Oct 30 01:42:15.338159 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.337965    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-workload-socket\") pod \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") "
Oct 30 01:42:15.338159 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.338026    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-kube-api-access-67hhf\" (UniqueName: \"kubernetes.io/projected/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-kube-api-access-67hhf\") pod \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") "
Oct 30 01:42:15.338308 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.338184    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-token\" (UniqueName: \"kubernetes.io/projected/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-token\") pod \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") "
Oct 30 01:42:15.338308 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.338237    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-workload-certs\") pod \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") "
Oct 30 01:42:15.338308 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.338276    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-credential-socket" pod "6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" (UID: "6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:42:15.338308 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.338306    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-podinfo\") pod \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") "
Oct 30 01:42:15.338698 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.338668    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-workload-certs" pod "6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" (UID: "6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:42:15.339185 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.339157    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istiod-ca-cert\") pod \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") "
Oct 30 01:42:15.339461 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.339200    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-envoy\") pod \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") "
Oct 30 01:42:15.339461 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.339277    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-data\") pod \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\" (UID: \"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5\") "
Oct 30 01:42:15.339766 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.339738    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-data" pod "6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" (UID: "6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:42:15.339876 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.339827    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-workload-socket" pod "6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" (UID: "6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:42:15.339954 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.339937    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:15.340015 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.339959    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:15.340015 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.339974    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:15.340170 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.340147    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istiod-ca-cert" pod "6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" (UID: "6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:42:15.345795 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.345762    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-envoy" pod "6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" (UID: "6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:42:15.345916 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.345829    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-kube-api-access-67hhf" pod "6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" (UID: "6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5"). InnerVolumeSpecName "kube-api-access-67hhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:42:15.345996 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.345922    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-podinfo" pod "6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" (UID: "6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:42:15.346333 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.346307    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-token" pod "6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" (UID: "6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:42:15.441333 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.441298    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:15.441333 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.441327    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:15.441333 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.441337    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:15.441577 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.441346    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67hhf\" (UniqueName: \"kubernetes.io/projected/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-kube-api-access-67hhf\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:15.441577 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.441357    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:15.441577 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.441365    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:42:15.711763 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.711728    4864 generic.go:356] "Generic (PLEG): container finished" podID="6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" containerID="141e1d76ee6cf0f0238cf91bbe5fed92c12e621a7ea474ad428dc6f2cdbff24f" exitCode=0
Oct 30 01:42:15.711939 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.711779    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"
Oct 30 01:42:15.711939 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.711808    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf" event={"ID":"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5","Type":"ContainerDied","Data":"141e1d76ee6cf0f0238cf91bbe5fed92c12e621a7ea474ad428dc6f2cdbff24f"}
Oct 30 01:42:15.711939 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.711871    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf" event={"ID":"6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5","Type":"ContainerDied","Data":"156542469ecac54aa8f4d61d3466b39d7da4120f87095704f12b65f0c6cb8bf3"}
Oct 30 01:42:15.711939 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.711898    4864 scope.go:122] "RemoveContainer" containerID="141e1d76ee6cf0f0238cf91bbe5fed92c12e621a7ea474ad428dc6f2cdbff24f"
Oct 30 01:42:15.723807 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.723783    4864 scope.go:122] "RemoveContainer" containerID="435baf638e3d19d13bf699c500285b51717ecba2bde80a7fefce4837a65ba236"
Oct 30 01:42:15.733777 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.733754    4864 scope.go:122] "RemoveContainer" containerID="141e1d76ee6cf0f0238cf91bbe5fed92c12e621a7ea474ad428dc6f2cdbff24f"
Oct 30 01:42:15.734038 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:42:15.734019    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"141e1d76ee6cf0f0238cf91bbe5fed92c12e621a7ea474ad428dc6f2cdbff24f\": container with ID starting with 141e1d76ee6cf0f0238cf91bbe5fed92c12e621a7ea474ad428dc6f2cdbff24f not found: ID does not exist" containerID="141e1d76ee6cf0f0238cf91bbe5fed92c12e621a7ea474ad428dc6f2cdbff24f"
Oct 30 01:42:15.734105 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.734047    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141e1d76ee6cf0f0238cf91bbe5fed92c12e621a7ea474ad428dc6f2cdbff24f"} err="failed to get container status \"141e1d76ee6cf0f0238cf91bbe5fed92c12e621a7ea474ad428dc6f2cdbff24f\": rpc error: code = NotFound desc = could not find container \"141e1d76ee6cf0f0238cf91bbe5fed92c12e621a7ea474ad428dc6f2cdbff24f\": container with ID starting with 141e1d76ee6cf0f0238cf91bbe5fed92c12e621a7ea474ad428dc6f2cdbff24f not found: ID does not exist"
Oct 30 01:42:15.734105 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.734065    4864 scope.go:122] "RemoveContainer" containerID="435baf638e3d19d13bf699c500285b51717ecba2bde80a7fefce4837a65ba236"
Oct 30 01:42:15.734303 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:42:15.734289    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"435baf638e3d19d13bf699c500285b51717ecba2bde80a7fefce4837a65ba236\": container with ID starting with 435baf638e3d19d13bf699c500285b51717ecba2bde80a7fefce4837a65ba236 not found: ID does not exist" containerID="435baf638e3d19d13bf699c500285b51717ecba2bde80a7fefce4837a65ba236"
Oct 30 01:42:15.734346 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.734305    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"435baf638e3d19d13bf699c500285b51717ecba2bde80a7fefce4837a65ba236"} err="failed to get container status \"435baf638e3d19d13bf699c500285b51717ecba2bde80a7fefce4837a65ba236\": rpc error: code = NotFound desc = could not find container \"435baf638e3d19d13bf699c500285b51717ecba2bde80a7fefce4837a65ba236\": container with ID starting with 435baf638e3d19d13bf699c500285b51717ecba2bde80a7fefce4837a65ba236 not found: ID does not exist"
Oct 30 01:42:15.761566 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.761535    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"]
Oct 30 01:42:15.771743 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.771717    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-bd287f-openshift-default-575f554785-gpglf"]
Oct 30 01:42:15.794948 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:42:15.794919    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" path="/var/lib/kubelet/pods/6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5/volumes"
Oct 30 01:42:19.040794 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:42:19.040761    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:42:49.039824 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:42:49.039783    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:43:09.779517 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:09.779478    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl"]
Oct 30 01:43:14.780632 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:14.780563    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl" podUID="910c155f-e976-4ae1-bb59-7aa5cb000392" containerName="payload-pre-processing" containerID="cri-o://e6f820e7458982caaef30425d57134ec5ae39c66ae210394725dd4abb7507e31" gracePeriod=25
Oct 30 01:43:14.905994 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:14.905954    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl" podUID="910c155f-e976-4ae1-bb59-7aa5cb000392" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:43:17.916143 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.916103    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"]
Oct 30 01:43:17.916783 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.916768    4864 cpu_manager.go:414] "containerMap: removing container" podUID="6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" containerName="istio-proxy"
Oct 30 01:43:17.916829 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.916785    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" containerName="istio-proxy"
Oct 30 01:43:17.916829 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.916799    4864 cpu_manager.go:414] "containerMap: removing container" podUID="e6ae4638-c9e9-4d20-b699-feb3208662a2" containerName="istio-proxy"
Oct 30 01:43:17.916829 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.916805    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ae4638-c9e9-4d20-b699-feb3208662a2" containerName="istio-proxy"
Oct 30 01:43:17.916829 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.916815    4864 cpu_manager.go:414] "containerMap: removing container" podUID="e6ae4638-c9e9-4d20-b699-feb3208662a2" containerName="istio-proxy"
Oct 30 01:43:17.916829 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.916820    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ae4638-c9e9-4d20-b699-feb3208662a2" containerName="istio-proxy"
Oct 30 01:43:17.917044 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.916835    4864 cpu_manager.go:414] "containerMap: removing container" podUID="2fa04251-2153-46ef-b2fe-0b3583cfbaf9" containerName="istio-proxy"
Oct 30 01:43:17.917044 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.916840    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa04251-2153-46ef-b2fe-0b3583cfbaf9" containerName="istio-proxy"
Oct 30 01:43:17.917044 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.916850    4864 cpu_manager.go:414] "containerMap: removing container" podUID="6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" containerName="istio-proxy"
Oct 30 01:43:17.917044 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.916855    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" containerName="istio-proxy"
Oct 30 01:43:17.917044 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.916865    4864 cpu_manager.go:414] "containerMap: removing container" podUID="2c5848bd-c585-46df-93d6-d63b46d80c3b" containerName="istio-proxy"
Oct 30 01:43:17.917044 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.916870    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c5848bd-c585-46df-93d6-d63b46d80c3b" containerName="istio-proxy"
Oct 30 01:43:17.917044 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.916878    4864 cpu_manager.go:414] "containerMap: removing container" podUID="db3e7716-143c-4e34-8d22-daae99a75109" containerName="istio-proxy"
Oct 30 01:43:17.917044 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.916883    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db3e7716-143c-4e34-8d22-daae99a75109" containerName="istio-proxy"
Oct 30 01:43:17.917044 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.916898    4864 cpu_manager.go:414] "containerMap: removing container" podUID="916074d0-6586-417a-baf6-df529fdd58b8" containerName="main"
Oct 30 01:43:17.917044 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.916903    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916074d0-6586-417a-baf6-df529fdd58b8" containerName="main"
Oct 30 01:43:17.917044 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.916920    4864 cpu_manager.go:414] "containerMap: removing container" podUID="056d952a-9dde-494f-9504-7e104751184b" containerName="main"
Oct 30 01:43:17.917044 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.916924    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="056d952a-9dde-494f-9504-7e104751184b" containerName="main"
Oct 30 01:43:17.917044 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.916932    4864 cpu_manager.go:414] "containerMap: removing container" podUID="3aa394cb-eba6-4e15-9a0e-523e2379f7ac" containerName="istio-proxy"
Oct 30 01:43:17.917044 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.916937    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa394cb-eba6-4e15-9a0e-523e2379f7ac" containerName="istio-proxy"
Oct 30 01:43:17.917044 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.916945    4864 cpu_manager.go:414] "containerMap: removing container" podUID="0d79aa0b-3be0-4837-8182-e1e940a4d73a" containerName="istio-proxy"
Oct 30 01:43:17.917044 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.916949    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d79aa0b-3be0-4837-8182-e1e940a4d73a" containerName="istio-proxy"
Oct 30 01:43:17.917044 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.917037    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="056d952a-9dde-494f-9504-7e104751184b" containerName="main"
Oct 30 01:43:17.917044 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.917046    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa394cb-eba6-4e15-9a0e-523e2379f7ac" containerName="istio-proxy"
Oct 30 01:43:17.917745 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.917053    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d79aa0b-3be0-4837-8182-e1e940a4d73a" containerName="istio-proxy"
Oct 30 01:43:17.917745 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.917060    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" containerName="istio-proxy"
Oct 30 01:43:17.917745 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.917065    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ae4638-c9e9-4d20-b699-feb3208662a2" containerName="istio-proxy"
Oct 30 01:43:17.917745 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.917087    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ae4638-c9e9-4d20-b699-feb3208662a2" containerName="istio-proxy"
Oct 30 01:43:17.917745 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.917095    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fa04251-2153-46ef-b2fe-0b3583cfbaf9" containerName="istio-proxy"
Oct 30 01:43:17.917745 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.917103    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c5848bd-c585-46df-93d6-d63b46d80c3b" containerName="istio-proxy"
Oct 30 01:43:17.917745 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.917110    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="db3e7716-143c-4e34-8d22-daae99a75109" containerName="istio-proxy"
Oct 30 01:43:17.917745 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.917117    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="916074d0-6586-417a-baf6-df529fdd58b8" containerName="main"
Oct 30 01:43:17.920871 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.920850    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:17.933954 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.933926    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6aa9aa-openshift-default-dockercfg-wrslw\""
Oct 30 01:43:17.948353 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.948324    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"]
Oct 30 01:43:17.991748 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.991699    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwq7j\" (UniqueName: \"kubernetes.io/projected/2d617aa1-07b4-400c-8a69-f356e0ae7500-kube-api-access-gwq7j\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:17.991927 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.991760    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-podinfo\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:17.992155 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.992130    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-envoy\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:17.992212 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.992193    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-token\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:17.992350 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.992323    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-workload-certs\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:17.992461 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.992441    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-credential-socket\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:17.992526 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.992505    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-data\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:17.992581 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.992554    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-workload-socket\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:17.992660 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:17.992640    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d617aa1-07b4-400c-8a69-f356e0ae7500-istiod-ca-cert\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:18.094691 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:18.094652    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-data\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:18.094876 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:18.094703    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-workload-socket\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:18.094876 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:18.094738    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d617aa1-07b4-400c-8a69-f356e0ae7500-istiod-ca-cert\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:18.094876 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:18.094815    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwq7j\" (UniqueName: \"kubernetes.io/projected/2d617aa1-07b4-400c-8a69-f356e0ae7500-kube-api-access-gwq7j\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:18.094876 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:18.094838    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-podinfo\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:18.094876 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:18.094871    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-envoy\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:18.095186 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:18.094909    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-token\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:18.095186 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:18.094991    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-workload-certs\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:18.095186 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:18.095060    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-data\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:18.095186 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:18.095086    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-credential-socket\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:18.095413 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:18.095328    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-workload-socket\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:18.095477 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:18.095417    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-workload-certs\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:18.095477 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:18.095466    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-credential-socket\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:18.095609 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:18.095583    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d617aa1-07b4-400c-8a69-f356e0ae7500-istiod-ca-cert\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:18.097460 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:18.097441    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-envoy\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:18.097775 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:18.097758    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-podinfo\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:18.116232 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:18.116201    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-token\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:18.116401 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:18.116383    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwq7j\" (UniqueName: \"kubernetes.io/projected/2d617aa1-07b4-400c-8a69-f356e0ae7500-kube-api-access-gwq7j\") pod \"e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") " pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:18.237404 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:18.237368    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:18.557303 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:18.557275    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"]
Oct 30 01:43:18.558570 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:43:18.558544    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d617aa1_07b4_400c_8a69_f356e0ae7500.slice/crio-cbf884e74d8abfa75f98f98ec424cd0ead827e1a1cace81a435559cfb5cf950b WatchSource:0}: Error finding container cbf884e74d8abfa75f98f98ec424cd0ead827e1a1cace81a435559cfb5cf950b: Status 404 returned error can't find the container with id cbf884e74d8abfa75f98f98ec424cd0ead827e1a1cace81a435559cfb5cf950b
Oct 30 01:43:18.561468 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:18.561435    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:43:18.561553 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:18.561501    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:43:18.561553 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:18.561533    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:43:19.042003 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:43:19.041955    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:43:19.214191 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:19.214150    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs" event={"ID":"2d617aa1-07b4-400c-8a69-f356e0ae7500","Type":"ContainerStarted","Data":"ec65824927ad61dfc61622762eae2b759b787abe670700a977d31d6fd8f7bab3"}
Oct 30 01:43:19.214396 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:19.214223    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs" event={"ID":"2d617aa1-07b4-400c-8a69-f356e0ae7500","Type":"ContainerStarted","Data":"cbf884e74d8abfa75f98f98ec424cd0ead827e1a1cace81a435559cfb5cf950b"}
Oct 30 01:43:19.237930 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:19.237899    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:19.239860 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:19.239840    4864 patch_prober.go:29] interesting pod/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 30 01:43:19.239946 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:19.239884    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs" podUID="2d617aa1-07b4-400c-8a69-f356e0ae7500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 30 01:43:19.269083 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:19.269016    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs" podStartSLOduration=2.269002045 podStartE2EDuration="2.269002045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:19.266351121 +0000 UTC m=+3711.604453032" watchObservedRunningTime="2026-10-30 01:43:19.269002045 +0000 UTC m=+3711.607103955"
Oct 30 01:43:20.238170 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:20.238134    4864 patch_prober.go:29] interesting pod/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 30 01:43:20.238688 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:20.238187    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs" podUID="2d617aa1-07b4-400c-8a69-f356e0ae7500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 30 01:43:21.238234 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:21.238189    4864 patch_prober.go:29] interesting pod/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 30 01:43:21.238622 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:21.238253    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs" podUID="2d617aa1-07b4-400c-8a69-f356e0ae7500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 30 01:43:22.238029 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:22.237993    4864 patch_prober.go:29] interesting pod/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 30 01:43:22.238261 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:22.238051    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs" podUID="2d617aa1-07b4-400c-8a69-f356e0ae7500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 30 01:43:23.238328 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:23.238291    4864 patch_prober.go:29] interesting pod/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 30 01:43:23.238835 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:23.238344    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs" podUID="2d617aa1-07b4-400c-8a69-f356e0ae7500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 30 01:43:24.237779 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:24.237746    4864 patch_prober.go:29] interesting pod/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 30 01:43:24.238009 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:24.237798    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs" podUID="2d617aa1-07b4-400c-8a69-f356e0ae7500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 30 01:43:24.905705 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:24.905669    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl" podUID="910c155f-e976-4ae1-bb59-7aa5cb000392" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:43:25.238341 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:25.238309    4864 patch_prober.go:29] interesting pod/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 30 01:43:25.238561 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:25.238380    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs" podUID="2d617aa1-07b4-400c-8a69-f356e0ae7500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 30 01:43:26.243417 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:26.243384    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:26.243874 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:26.243852    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:26.244623 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:26.244600    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:43:30.677449 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.677414    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"]
Oct 30 01:43:30.678429 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.678408    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c58c64b-ffc4-4ef6-b1fd-b60360eb37b5" containerName="istio-proxy"
Oct 30 01:43:30.686545 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.686521    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.700560 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.700538    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5457f2-openshift-default-dockercfg-dlvjl\""
Oct 30 01:43:30.716979 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.716945    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"]
Oct 30 01:43:30.814526 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.814494    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-podinfo\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.814703 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.814542    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-envoy\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.814703 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.814640    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jttlh\" (UniqueName: \"kubernetes.io/projected/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-kube-api-access-jttlh\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.814703 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.814677    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-token\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.814873 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.814752    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-data\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.814873 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.814852    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-credential-socket\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.814976 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.814927    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istiod-ca-cert\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.815021 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.814982    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-workload-socket\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.815105 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.815066    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-workload-certs\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.919984 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.917501    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-data\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.919984 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.917582    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-credential-socket\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.919984 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.917677    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istiod-ca-cert\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.919984 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.917742    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-workload-socket\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.919984 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.917873    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-workload-certs\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.919984 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.917999    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-podinfo\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.919984 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.918043    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-envoy\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.919984 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.918114    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jttlh\" (UniqueName: \"kubernetes.io/projected/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-kube-api-access-jttlh\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.919984 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.918180    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-token\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.919984 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.918939    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-data\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.919984 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.918949    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-credential-socket\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.919984 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.919538    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-workload-certs\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.919984 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.919686    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istiod-ca-cert\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.919984 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.919923    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-workload-socket\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.933865 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.933761    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-envoy\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.934277 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.934246    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-podinfo\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.944194 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.944136    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-token\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:30.944581 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:30.944556    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jttlh\" (UniqueName: \"kubernetes.io/projected/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-kube-api-access-jttlh\") pod \"e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") " pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:31.000528 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:31.000490    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:31.321183 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:31.321153    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"]
Oct 30 01:43:31.322382 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:43:31.322358    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f94d41_d691_4cdc_a24b_fc976bc9e1c8.slice/crio-e8e2314dfa13fa9282a40780852bc762ccee5f9ba19f8ae396af3e317d4c145c WatchSource:0}: Error finding container e8e2314dfa13fa9282a40780852bc762ccee5f9ba19f8ae396af3e317d4c145c: Status 404 returned error can't find the container with id e8e2314dfa13fa9282a40780852bc762ccee5f9ba19f8ae396af3e317d4c145c
Oct 30 01:43:31.325387 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:31.325352    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:43:31.325482 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:31.325420    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:43:31.325482 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:31.325449    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:43:32.308142 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:32.308101    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85" event={"ID":"31f94d41-d691-4cdc-a24b-fc976bc9e1c8","Type":"ContainerStarted","Data":"86e6aa31c29cb1847eb0cff35f1e78063a74764aaa4b2c347a3bb959cc1d5498"}
Oct 30 01:43:32.308616 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:32.308168    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85" event={"ID":"31f94d41-d691-4cdc-a24b-fc976bc9e1c8","Type":"ContainerStarted","Data":"e8e2314dfa13fa9282a40780852bc762ccee5f9ba19f8ae396af3e317d4c145c"}
Oct 30 01:43:33.001531 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:33.001497    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:33.003755 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:33.003727    4864 patch_prober.go:29] interesting pod/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 01:43:33.003901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:33.003771    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85" podUID="31f94d41-d691-4cdc-a24b-fc976bc9e1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 01:43:34.000986 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:34.000952    4864 patch_prober.go:29] interesting pod/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 01:43:34.001408 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:34.001010    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85" podUID="31f94d41-d691-4cdc-a24b-fc976bc9e1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 01:43:34.905606 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:43:34.905580    4864 logging.go:55] [core] [Channel #6839 SubChannel #6840]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.154:50052", ServerName: "10.132.0.154:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.154:50052: connect: connection refused"
Oct 30 01:43:35.029751 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:35.027404    4864 patch_prober.go:29] interesting pod/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 01:43:35.029751 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:35.027463    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85" podUID="31f94d41-d691-4cdc-a24b-fc976bc9e1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 01:43:35.138103 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:35.138046    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl"
Oct 30 01:43:35.185100 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:35.184501    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85" podStartSLOduration=5.184479498 podStartE2EDuration="5.184479498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:32.385200835 +0000 UTC m=+3724.723302746" watchObservedRunningTime="2026-10-30 01:43:35.184479498 +0000 UTC m=+3727.522581409"
Oct 30 01:43:35.231818 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:35.231787    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/910c155f-e976-4ae1-bb59-7aa5cb000392-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/910c155f-e976-4ae1-bb59-7aa5cb000392-plugins-config-volume\") pod \"910c155f-e976-4ae1-bb59-7aa5cb000392\" (UID: \"910c155f-e976-4ae1-bb59-7aa5cb000392\") "
Oct 30 01:43:35.232014 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:35.231931    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/910c155f-e976-4ae1-bb59-7aa5cb000392-kube-api-access-fqvz9\" (UniqueName: \"kubernetes.io/projected/910c155f-e976-4ae1-bb59-7aa5cb000392-kube-api-access-fqvz9\") pod \"910c155f-e976-4ae1-bb59-7aa5cb000392\" (UID: \"910c155f-e976-4ae1-bb59-7aa5cb000392\") "
Oct 30 01:43:35.232282 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:35.232252    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/910c155f-e976-4ae1-bb59-7aa5cb000392-plugins-config-volume" pod "910c155f-e976-4ae1-bb59-7aa5cb000392" (UID: "910c155f-e976-4ae1-bb59-7aa5cb000392"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:43:35.232668 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:35.232644    4864 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/910c155f-e976-4ae1-bb59-7aa5cb000392-plugins-config-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:43:35.234284 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:35.234254    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/910c155f-e976-4ae1-bb59-7aa5cb000392-kube-api-access-fqvz9" pod "910c155f-e976-4ae1-bb59-7aa5cb000392" (UID: "910c155f-e976-4ae1-bb59-7aa5cb000392"). InnerVolumeSpecName "kube-api-access-fqvz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:43:35.333758 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:35.333719    4864 generic.go:356] "Generic (PLEG): container finished" podID="910c155f-e976-4ae1-bb59-7aa5cb000392" containerID="e6f820e7458982caaef30425d57134ec5ae39c66ae210394725dd4abb7507e31" exitCode=0
Oct 30 01:43:35.333946 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:35.333761    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl"
Oct 30 01:43:35.333946 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:35.333801    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl" event={"ID":"910c155f-e976-4ae1-bb59-7aa5cb000392","Type":"ContainerDied","Data":"e6f820e7458982caaef30425d57134ec5ae39c66ae210394725dd4abb7507e31"}
Oct 30 01:43:35.333946 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:35.333838    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqvz9\" (UniqueName: \"kubernetes.io/projected/910c155f-e976-4ae1-bb59-7aa5cb000392-kube-api-access-fqvz9\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:43:35.333946 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:35.333857    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl" event={"ID":"910c155f-e976-4ae1-bb59-7aa5cb000392","Type":"ContainerDied","Data":"ab2cdb85cb13ab15b7d77ccc827e076359c28874ab199cd8caa5c34ea70644c9"}
Oct 30 01:43:35.333946 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:35.333884    4864 scope.go:122] "RemoveContainer" containerID="e6f820e7458982caaef30425d57134ec5ae39c66ae210394725dd4abb7507e31"
Oct 30 01:43:35.344638 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:35.344618    4864 scope.go:122] "RemoveContainer" containerID="e6f820e7458982caaef30425d57134ec5ae39c66ae210394725dd4abb7507e31"
Oct 30 01:43:35.344916 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:43:35.344898    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f820e7458982caaef30425d57134ec5ae39c66ae210394725dd4abb7507e31\": container with ID starting with e6f820e7458982caaef30425d57134ec5ae39c66ae210394725dd4abb7507e31 not found: ID does not exist" containerID="e6f820e7458982caaef30425d57134ec5ae39c66ae210394725dd4abb7507e31"
Oct 30 01:43:35.344995 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:35.344923    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f820e7458982caaef30425d57134ec5ae39c66ae210394725dd4abb7507e31"} err="failed to get container status \"e6f820e7458982caaef30425d57134ec5ae39c66ae210394725dd4abb7507e31\": rpc error: code = NotFound desc = could not find container \"e6f820e7458982caaef30425d57134ec5ae39c66ae210394725dd4abb7507e31\": container with ID starting with e6f820e7458982caaef30425d57134ec5ae39c66ae210394725dd4abb7507e31 not found: ID does not exist"
Oct 30 01:43:35.402388 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:35.402299    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl"]
Oct 30 01:43:35.409718 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:35.409689    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl"]
Oct 30 01:43:35.794514 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:35.794477    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="910c155f-e976-4ae1-bb59-7aa5cb000392" path="/var/lib/kubelet/pods/910c155f-e976-4ae1-bb59-7aa5cb000392/volumes"
Oct 30 01:43:35.906270 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:35.906228    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ab0a2c-54f5b465b7-b6lfl" podUID="910c155f-e976-4ae1-bb59-7aa5cb000392" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.154:50052\" within 1s: context deadline exceeded"
Oct 30 01:43:36.001745 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:36.001712    4864 patch_prober.go:29] interesting pod/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 01:43:36.001923 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:36.001765    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85" podUID="31f94d41-d691-4cdc-a24b-fc976bc9e1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 01:43:37.001433 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:37.001399    4864 patch_prober.go:29] interesting pod/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 01:43:37.001838 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:37.001471    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85" podUID="31f94d41-d691-4cdc-a24b-fc976bc9e1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 01:43:38.006955 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:38.006926    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:38.007446 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:38.007423    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:38.008182 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:38.008158    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:43:40.843033 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:40.842944    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"]
Oct 30 01:43:49.041662 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:43:49.041625    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:43:55.865362 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:55.865312    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb" podUID="20a2e5ac-0716-4f26-bc37-b0682e5c8988" containerName="main" containerID="cri-o://7d926e8f34cda5f906076181841f08172b3f3f3dd69089f8ae60947708e0138a" gracePeriod=45
Oct 30 01:43:56.150391 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.150366    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:43:56.279408 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.279375    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20a2e5ac-0716-4f26-bc37-b0682e5c8988-tls-certs\" (UniqueName: \"kubernetes.io/secret/20a2e5ac-0716-4f26-bc37-b0682e5c8988-tls-certs\") pod \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\" (UID: \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\") "
Oct 30 01:43:56.279574 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.279420    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a2e5ac-0716-4f26-bc37-b0682e5c8988-kube-api-access-px6gl\" (UniqueName: \"kubernetes.io/projected/20a2e5ac-0716-4f26-bc37-b0682e5c8988-kube-api-access-px6gl\") pod \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\" (UID: \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\") "
Oct 30 01:43:56.279574 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.279473    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-home\" (UniqueName: \"kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-home\") pod \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\" (UID: \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\") "
Oct 30 01:43:56.279574 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.279496    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-tmp-dir\") pod \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\" (UID: \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\") "
Oct 30 01:43:56.279717 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.279616    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-model-cache\") pod \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\" (UID: \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\") "
Oct 30 01:43:56.279717 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.279647    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-dshm\") pod \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\" (UID: \"20a2e5ac-0716-4f26-bc37-b0682e5c8988\") "
Oct 30 01:43:56.279836 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.279731    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-home" pod "20a2e5ac-0716-4f26-bc37-b0682e5c8988" (UID: "20a2e5ac-0716-4f26-bc37-b0682e5c8988"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:43:56.279836 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.279772    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-tmp-dir" pod "20a2e5ac-0716-4f26-bc37-b0682e5c8988" (UID: "20a2e5ac-0716-4f26-bc37-b0682e5c8988"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:43:56.279951 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.279877    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-model-cache" pod "20a2e5ac-0716-4f26-bc37-b0682e5c8988" (UID: "20a2e5ac-0716-4f26-bc37-b0682e5c8988"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:43:56.280386 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.280362    4864 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-model-cache\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:43:56.280386 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.280386    4864 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-home\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:43:56.280548 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.280420    4864 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-tmp-dir\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:43:56.281953 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.281930    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-dshm" pod "20a2e5ac-0716-4f26-bc37-b0682e5c8988" (UID: "20a2e5ac-0716-4f26-bc37-b0682e5c8988"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:43:56.282065 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.281964    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a2e5ac-0716-4f26-bc37-b0682e5c8988-kube-api-access-px6gl" pod "20a2e5ac-0716-4f26-bc37-b0682e5c8988" (UID: "20a2e5ac-0716-4f26-bc37-b0682e5c8988"). InnerVolumeSpecName "kube-api-access-px6gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:43:56.282440 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.282422    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a2e5ac-0716-4f26-bc37-b0682e5c8988-tls-certs" pod "20a2e5ac-0716-4f26-bc37-b0682e5c8988" (UID: "20a2e5ac-0716-4f26-bc37-b0682e5c8988"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:43:56.382456 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.382371    4864 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a2e5ac-0716-4f26-bc37-b0682e5c8988-dshm\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:43:56.382456 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.382404    4864 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a2e5ac-0716-4f26-bc37-b0682e5c8988-tls-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:43:56.382456 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.382415    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px6gl\" (UniqueName: \"kubernetes.io/projected/20a2e5ac-0716-4f26-bc37-b0682e5c8988-kube-api-access-px6gl\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:43:56.495976 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.495938    4864 generic.go:356] "Generic (PLEG): container finished" podID="20a2e5ac-0716-4f26-bc37-b0682e5c8988" containerID="7d926e8f34cda5f906076181841f08172b3f3f3dd69089f8ae60947708e0138a" exitCode=0
Oct 30 01:43:56.495976 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.495987    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"
Oct 30 01:43:56.496259 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.496024    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb" event={"ID":"20a2e5ac-0716-4f26-bc37-b0682e5c8988","Type":"ContainerDied","Data":"7d926e8f34cda5f906076181841f08172b3f3f3dd69089f8ae60947708e0138a"}
Oct 30 01:43:56.496259 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.496106    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb" event={"ID":"20a2e5ac-0716-4f26-bc37-b0682e5c8988","Type":"ContainerDied","Data":"1edd8731812703280f457a63dd0ee704623059b6b58442cac6346d9bfd724434"}
Oct 30 01:43:56.496259 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.496132    4864 scope.go:122] "RemoveContainer" containerID="7d926e8f34cda5f906076181841f08172b3f3f3dd69089f8ae60947708e0138a"
Oct 30 01:43:56.509232 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.509215    4864 scope.go:122] "RemoveContainer" containerID="7d926e8f34cda5f906076181841f08172b3f3f3dd69089f8ae60947708e0138a"
Oct 30 01:43:56.509517 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:43:56.509496    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d926e8f34cda5f906076181841f08172b3f3f3dd69089f8ae60947708e0138a\": container with ID starting with 7d926e8f34cda5f906076181841f08172b3f3f3dd69089f8ae60947708e0138a not found: ID does not exist" containerID="7d926e8f34cda5f906076181841f08172b3f3f3dd69089f8ae60947708e0138a"
Oct 30 01:43:56.509573 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.509525    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d926e8f34cda5f906076181841f08172b3f3f3dd69089f8ae60947708e0138a"} err="failed to get container status \"7d926e8f34cda5f906076181841f08172b3f3f3dd69089f8ae60947708e0138a\": rpc error: code = NotFound desc = could not find container \"7d926e8f34cda5f906076181841f08172b3f3f3dd69089f8ae60947708e0138a\": container with ID starting with 7d926e8f34cda5f906076181841f08172b3f3f3dd69089f8ae60947708e0138a not found: ID does not exist"
Oct 30 01:43:56.557199 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.557174    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"]
Oct 30 01:43:56.579578 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:56.579546    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b29d4a/test-model-b29d4a-kserve-56c88dd54c-km2lb"]
Oct 30 01:43:57.795637 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:43:57.795603    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20a2e5ac-0716-4f26-bc37-b0682e5c8988" path="/var/lib/kubelet/pods/20a2e5ac-0716-4f26-bc37-b0682e5c8988/volumes"
Oct 30 01:44:03.369208 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:03.369172    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"]
Oct 30 01:44:07.431776 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.431741    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"]
Oct 30 01:44:07.432441 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.432425    4864 cpu_manager.go:414] "containerMap: removing container" podUID="910c155f-e976-4ae1-bb59-7aa5cb000392" containerName="payload-pre-processing"
Oct 30 01:44:07.432499 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.432442    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="910c155f-e976-4ae1-bb59-7aa5cb000392" containerName="payload-pre-processing"
Oct 30 01:44:07.432499 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.432463    4864 cpu_manager.go:414] "containerMap: removing container" podUID="20a2e5ac-0716-4f26-bc37-b0682e5c8988" containerName="main"
Oct 30 01:44:07.432499 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.432468    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a2e5ac-0716-4f26-bc37-b0682e5c8988" containerName="main"
Oct 30 01:44:07.432626 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.432616    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="910c155f-e976-4ae1-bb59-7aa5cb000392" containerName="payload-pre-processing"
Oct 30 01:44:07.432661 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.432631    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="20a2e5ac-0716-4f26-bc37-b0682e5c8988" containerName="main"
Oct 30 01:44:07.436355 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.436338    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.464042 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.464010    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"]
Oct 30 01:44:07.538526 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.538485    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f9vf\" (UniqueName: \"kubernetes.io/projected/9de67abe-4f7c-441f-9e3e-cf34c74068d9-kube-api-access-7f9vf\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.538713 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.538543    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.538713 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.538627    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.538822 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.538747    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.538822 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.538809    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.538933 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.538832    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.538933 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.538874    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.539066 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.539051    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.539167 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.539152    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.640962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.640926    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f9vf\" (UniqueName: \"kubernetes.io/projected/9de67abe-4f7c-441f-9e3e-cf34c74068d9-kube-api-access-7f9vf\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.641179 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.640975    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.641179 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.641011    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.641179 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.641133    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.641349 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.641203    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.641349 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.641238    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.641349 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.641261    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.641508 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.641361    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.641508 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.641440    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.641508 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.641450    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.641508 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.641479    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.641813 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.641792    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.641902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.641792    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.642102 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.642059    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.644220 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.644197    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.644322 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.644306    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.665782 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.665755    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.666147 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.666127    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f9vf\" (UniqueName: \"kubernetes.io/projected/9de67abe-4f7c-441f-9e3e-cf34c74068d9-kube-api-access-7f9vf\") pod \"maas-default-gateway-openshift-default-fcb75c45f-89zld\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:07.751087 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:07.751050    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:08.094941 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:08.094914    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"]
Oct 30 01:44:08.097388 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:44:08.097359    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de67abe_4f7c_441f_9e3e_cf34c74068d9.slice/crio-aeebcc32137e2b837da9ced4300bbb54335067431fc4031bdb52dad68eda4705 WatchSource:0}: Error finding container aeebcc32137e2b837da9ced4300bbb54335067431fc4031bdb52dad68eda4705: Status 404 returned error can't find the container with id aeebcc32137e2b837da9ced4300bbb54335067431fc4031bdb52dad68eda4705
Oct 30 01:44:08.100513 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:08.100482    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:44:08.100597 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:08.100554    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:44:08.100597 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:08.100582    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:44:08.589766 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:08.589715    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld" event={"ID":"9de67abe-4f7c-441f-9e3e-cf34c74068d9","Type":"ContainerStarted","Data":"fa4d3a238a1c681ec10393a53e58be3242307bd7f9be38200f339e40cec41164"}
Oct 30 01:44:08.590255 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:08.589787    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld" event={"ID":"9de67abe-4f7c-441f-9e3e-cf34c74068d9","Type":"ContainerStarted","Data":"aeebcc32137e2b837da9ced4300bbb54335067431fc4031bdb52dad68eda4705"}
Oct 30 01:44:08.634182 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:08.634128    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld" podStartSLOduration=1.6341132329999999 podStartE2EDuration="1.634113233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:44:08.631453898 +0000 UTC m=+3760.969555808" watchObservedRunningTime="2026-10-30 01:44:08.634113233 +0000 UTC m=+3760.972215137"
Oct 30 01:44:08.751451 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:08.751419    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:09.753874 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:09.753835    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-89zld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 01:44:09.754305 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:09.753884    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld" podUID="9de67abe-4f7c-441f-9e3e-cf34c74068d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 01:44:10.097409 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:10.097304    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"]
Oct 30 01:44:10.097637 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:10.097546    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl" podUID="55134196-9552-406d-931e-b95123dcbd94" containerName="istio-proxy" containerID="cri-o://cc58e936ae5a72fb5121595962a6ab14e30e73449a9f4ead9ef808c8601f5cad" gracePeriod=30
Oct 30 01:44:10.751615 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:10.751580    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-89zld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 01:44:10.751780 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:10.751633    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld" podUID="9de67abe-4f7c-441f-9e3e-cf34c74068d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 01:44:11.751995 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:11.751959    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-89zld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 01:44:11.752542 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:11.752020    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld" podUID="9de67abe-4f7c-441f-9e3e-cf34c74068d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 01:44:12.751977 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:12.751943    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-89zld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 01:44:12.752176 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:12.751996    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld" podUID="9de67abe-4f7c-441f-9e3e-cf34c74068d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 01:44:13.752338 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:13.752307    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-89zld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 01:44:13.752756 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:13.752357    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld" podUID="9de67abe-4f7c-441f-9e3e-cf34c74068d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 01:44:14.751936 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:14.751894    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-89zld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 01:44:14.752128 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:14.751966    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld" podUID="9de67abe-4f7c-441f-9e3e-cf34c74068d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 01:44:15.385058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.384996    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:44:15.392703 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.392678    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55134196-9552-406d-931e-b95123dcbd94-istio-token\" (UniqueName: \"kubernetes.io/projected/55134196-9552-406d-931e-b95123dcbd94-istio-token\") pod \"55134196-9552-406d-931e-b95123dcbd94\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") "
Oct 30 01:44:15.392846 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.392733    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55134196-9552-406d-931e-b95123dcbd94-kube-api-access-8jght\" (UniqueName: \"kubernetes.io/projected/55134196-9552-406d-931e-b95123dcbd94-kube-api-access-8jght\") pod \"55134196-9552-406d-931e-b95123dcbd94\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") "
Oct 30 01:44:15.392846 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.392760    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-istio-envoy\") pod \"55134196-9552-406d-931e-b95123dcbd94\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") "
Oct 30 01:44:15.392846 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.392803    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-credential-socket\") pod \"55134196-9552-406d-931e-b95123dcbd94\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") "
Oct 30 01:44:15.393021 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.392958    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-istio-data\") pod \"55134196-9552-406d-931e-b95123dcbd94\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") "
Oct 30 01:44:15.393021 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.392980    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-workload-certs\") pod \"55134196-9552-406d-931e-b95123dcbd94\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") "
Oct 30 01:44:15.393161 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.393034    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-workload-socket\") pod \"55134196-9552-406d-931e-b95123dcbd94\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") "
Oct 30 01:44:15.393161 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.393088    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55134196-9552-406d-931e-b95123dcbd94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55134196-9552-406d-931e-b95123dcbd94-istiod-ca-cert\") pod \"55134196-9552-406d-931e-b95123dcbd94\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") "
Oct 30 01:44:15.393161 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.393136    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55134196-9552-406d-931e-b95123dcbd94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55134196-9552-406d-931e-b95123dcbd94-istio-podinfo\") pod \"55134196-9552-406d-931e-b95123dcbd94\" (UID: \"55134196-9552-406d-931e-b95123dcbd94\") "
Oct 30 01:44:15.393318 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.393183    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-credential-socket" pod "55134196-9552-406d-931e-b95123dcbd94" (UID: "55134196-9552-406d-931e-b95123dcbd94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:44:15.393318 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.393230    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-istio-data" pod "55134196-9552-406d-931e-b95123dcbd94" (UID: "55134196-9552-406d-931e-b95123dcbd94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:44:15.393422 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.393398    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-workload-socket" pod "55134196-9552-406d-931e-b95123dcbd94" (UID: "55134196-9552-406d-931e-b95123dcbd94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:44:15.393630 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.393506    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55134196-9552-406d-931e-b95123dcbd94-istiod-ca-cert" pod "55134196-9552-406d-931e-b95123dcbd94" (UID: "55134196-9552-406d-931e-b95123dcbd94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:44:15.393630 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.393593    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-workload-certs" pod "55134196-9552-406d-931e-b95123dcbd94" (UID: "55134196-9552-406d-931e-b95123dcbd94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:44:15.393907 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.393885    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:44:15.393988 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.393914    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:44:15.393988 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.393930    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:44:15.393988 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.393943    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55134196-9552-406d-931e-b95123dcbd94-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:44:15.393988 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.393958    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:44:15.395222 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.395189    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55134196-9552-406d-931e-b95123dcbd94-kube-api-access-8jght" pod "55134196-9552-406d-931e-b95123dcbd94" (UID: "55134196-9552-406d-931e-b95123dcbd94"). InnerVolumeSpecName "kube-api-access-8jght". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:44:15.395321 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.395257    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55134196-9552-406d-931e-b95123dcbd94-istio-token" pod "55134196-9552-406d-931e-b95123dcbd94" (UID: "55134196-9552-406d-931e-b95123dcbd94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:44:15.395483 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.395453    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-istio-envoy" pod "55134196-9552-406d-931e-b95123dcbd94" (UID: "55134196-9552-406d-931e-b95123dcbd94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:44:15.395564 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.395498    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55134196-9552-406d-931e-b95123dcbd94-istio-podinfo" pod "55134196-9552-406d-931e-b95123dcbd94" (UID: "55134196-9552-406d-931e-b95123dcbd94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:44:15.495983 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.495946    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55134196-9552-406d-931e-b95123dcbd94-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:44:15.495983 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.495986    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jght\" (UniqueName: \"kubernetes.io/projected/55134196-9552-406d-931e-b95123dcbd94-kube-api-access-8jght\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:44:15.496266 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.496002    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55134196-9552-406d-931e-b95123dcbd94-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:44:15.496266 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.496017    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55134196-9552-406d-931e-b95123dcbd94-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:44:15.642240 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.642144    4864 generic.go:356] "Generic (PLEG): container finished" podID="55134196-9552-406d-931e-b95123dcbd94" containerID="cc58e936ae5a72fb5121595962a6ab14e30e73449a9f4ead9ef808c8601f5cad" exitCode=0
Oct 30 01:44:15.642240 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.642196    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"
Oct 30 01:44:15.642240 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.642204    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl" event={"ID":"55134196-9552-406d-931e-b95123dcbd94","Type":"ContainerDied","Data":"cc58e936ae5a72fb5121595962a6ab14e30e73449a9f4ead9ef808c8601f5cad"}
Oct 30 01:44:15.642508 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.642268    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl" event={"ID":"55134196-9552-406d-931e-b95123dcbd94","Type":"ContainerDied","Data":"e71afea8d3d7b88470f1bfaf2fe7b5f60e9ba9dc69d0a5b8e2ab3e199372d8cf"}
Oct 30 01:44:15.642508 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.642297    4864 scope.go:122] "RemoveContainer" containerID="cc58e936ae5a72fb5121595962a6ab14e30e73449a9f4ead9ef808c8601f5cad"
Oct 30 01:44:15.654798 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.654781    4864 scope.go:122] "RemoveContainer" containerID="01ba5bdcf22c660eeb97c700ecae563ecf36e4ff93dd38f758d996d42b9a2957"
Oct 30 01:44:15.664669 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.664648    4864 scope.go:122] "RemoveContainer" containerID="cc58e936ae5a72fb5121595962a6ab14e30e73449a9f4ead9ef808c8601f5cad"
Oct 30 01:44:15.664941 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:44:15.664918    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc58e936ae5a72fb5121595962a6ab14e30e73449a9f4ead9ef808c8601f5cad\": container with ID starting with cc58e936ae5a72fb5121595962a6ab14e30e73449a9f4ead9ef808c8601f5cad not found: ID does not exist" containerID="cc58e936ae5a72fb5121595962a6ab14e30e73449a9f4ead9ef808c8601f5cad"
Oct 30 01:44:15.665022 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.664949    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc58e936ae5a72fb5121595962a6ab14e30e73449a9f4ead9ef808c8601f5cad"} err="failed to get container status \"cc58e936ae5a72fb5121595962a6ab14e30e73449a9f4ead9ef808c8601f5cad\": rpc error: code = NotFound desc = could not find container \"cc58e936ae5a72fb5121595962a6ab14e30e73449a9f4ead9ef808c8601f5cad\": container with ID starting with cc58e936ae5a72fb5121595962a6ab14e30e73449a9f4ead9ef808c8601f5cad not found: ID does not exist"
Oct 30 01:44:15.665022 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.664965    4864 scope.go:122] "RemoveContainer" containerID="01ba5bdcf22c660eeb97c700ecae563ecf36e4ff93dd38f758d996d42b9a2957"
Oct 30 01:44:15.665263 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:44:15.665243    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01ba5bdcf22c660eeb97c700ecae563ecf36e4ff93dd38f758d996d42b9a2957\": container with ID starting with 01ba5bdcf22c660eeb97c700ecae563ecf36e4ff93dd38f758d996d42b9a2957 not found: ID does not exist" containerID="01ba5bdcf22c660eeb97c700ecae563ecf36e4ff93dd38f758d996d42b9a2957"
Oct 30 01:44:15.665350 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.665269    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01ba5bdcf22c660eeb97c700ecae563ecf36e4ff93dd38f758d996d42b9a2957"} err="failed to get container status \"01ba5bdcf22c660eeb97c700ecae563ecf36e4ff93dd38f758d996d42b9a2957\": rpc error: code = NotFound desc = could not find container \"01ba5bdcf22c660eeb97c700ecae563ecf36e4ff93dd38f758d996d42b9a2957\": container with ID starting with 01ba5bdcf22c660eeb97c700ecae563ecf36e4ff93dd38f758d996d42b9a2957 not found: ID does not exist"
Oct 30 01:44:15.690435 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.690398    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"]
Oct 30 01:44:15.694144 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.694119    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ab0a2c-openshift-default-548cd8bb59-fh5nl"]
Oct 30 01:44:15.751700 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.751661    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-89zld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 01:44:15.751870 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.751730    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld" podUID="9de67abe-4f7c-441f-9e3e-cf34c74068d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 01:44:15.795632 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:15.795603    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55134196-9552-406d-931e-b95123dcbd94" path="/var/lib/kubelet/pods/55134196-9552-406d-931e-b95123dcbd94/volumes"
Oct 30 01:44:17.032853 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:17.032822    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:17.033578 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:17.033547    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:17.034646 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:17.034617    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:44:18.392463 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:18.392419    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs" podUID="28a7413b-beb4-4b9f-bc76-7b1c5497ce1d" containerName="main" containerID="cri-o://b98fb73783afe0ef43e59d1762ac0e2d688ff51d5f89c1b00c467d8936451faf" gracePeriod=45
Oct 30 01:44:18.673211 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:18.673176    4864 generic.go:356] "Generic (PLEG): container finished" podID="28a7413b-beb4-4b9f-bc76-7b1c5497ce1d" containerID="b98fb73783afe0ef43e59d1762ac0e2d688ff51d5f89c1b00c467d8936451faf" exitCode=0
Oct 30 01:44:18.673359 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:18.673250    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs" event={"ID":"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d","Type":"ContainerDied","Data":"b98fb73783afe0ef43e59d1762ac0e2d688ff51d5f89c1b00c467d8936451faf"}
Oct 30 01:44:18.673359 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:18.673300    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs" event={"ID":"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d","Type":"ContainerDied","Data":"1d15dd068ae9533f3aa92e17947131cc249c4cc201c5d92906e90e83413ac068"}
Oct 30 01:44:18.673359 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:18.673314    4864 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d15dd068ae9533f3aa92e17947131cc249c4cc201c5d92906e90e83413ac068"
Oct 30 01:44:18.683259 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:18.683237    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:44:18.743406 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:18.743372    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-tls-certs\" (UniqueName: \"kubernetes.io/secret/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-tls-certs\") pod \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\" (UID: \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\") "
Oct 30 01:44:18.743592 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:18.743441    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-model-cache\") pod \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\" (UID: \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\") "
Oct 30 01:44:18.743592 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:18.743489    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-tmp-dir\") pod \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\" (UID: \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\") "
Oct 30 01:44:18.743592 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:18.743560    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-dshm\") pod \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\" (UID: \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\") "
Oct 30 01:44:18.743766 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:18.743625    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-home\" (UniqueName: \"kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-home\") pod \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\" (UID: \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\") "
Oct 30 01:44:18.743766 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:18.743697    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-kube-api-access-s4kr8\" (UniqueName: \"kubernetes.io/projected/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-kube-api-access-s4kr8\") pod \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\" (UID: \"28a7413b-beb4-4b9f-bc76-7b1c5497ce1d\") "
Oct 30 01:44:18.744040 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:18.743985    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-home" pod "28a7413b-beb4-4b9f-bc76-7b1c5497ce1d" (UID: "28a7413b-beb4-4b9f-bc76-7b1c5497ce1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:44:18.744040 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:18.744009    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-tmp-dir" pod "28a7413b-beb4-4b9f-bc76-7b1c5497ce1d" (UID: "28a7413b-beb4-4b9f-bc76-7b1c5497ce1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:44:18.745144 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:18.744271    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-model-cache" pod "28a7413b-beb4-4b9f-bc76-7b1c5497ce1d" (UID: "28a7413b-beb4-4b9f-bc76-7b1c5497ce1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:44:18.745144 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:18.744809    4864 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-model-cache\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:44:18.745144 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:18.744832    4864 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-tmp-dir\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:44:18.745144 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:18.744845    4864 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-home\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:44:18.746270 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:18.746243    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-dshm" pod "28a7413b-beb4-4b9f-bc76-7b1c5497ce1d" (UID: "28a7413b-beb4-4b9f-bc76-7b1c5497ce1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:44:18.746387 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:18.746369    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-tls-certs" pod "28a7413b-beb4-4b9f-bc76-7b1c5497ce1d" (UID: "28a7413b-beb4-4b9f-bc76-7b1c5497ce1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:44:18.746493 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:18.746476    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-kube-api-access-s4kr8" pod "28a7413b-beb4-4b9f-bc76-7b1c5497ce1d" (UID: "28a7413b-beb4-4b9f-bc76-7b1c5497ce1d"). InnerVolumeSpecName "kube-api-access-s4kr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:44:18.846280 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:18.846249    4864 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-tls-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:44:18.846280 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:18.846274    4864 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-dshm\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:44:18.846280 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:18.846286    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4kr8\" (UniqueName: \"kubernetes.io/projected/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d-kube-api-access-s4kr8\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:44:19.044658 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:44:19.044597    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:44:19.680227 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:19.680193    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"
Oct 30 01:44:19.751893 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:19.751854    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"]
Oct 30 01:44:19.762427 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:19.762398    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5457f2/test-model-5457f2-kserve-98946549-gmzfs"]
Oct 30 01:44:19.820660 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:19.820629    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28a7413b-beb4-4b9f-bc76-7b1c5497ce1d" path="/var/lib/kubelet/pods/28a7413b-beb4-4b9f-bc76-7b1c5497ce1d/volumes"
Oct 30 01:44:22.458898 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.458864    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"]
Oct 30 01:44:22.459613 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.459594    4864 cpu_manager.go:414] "containerMap: removing container" podUID="55134196-9552-406d-931e-b95123dcbd94" containerName="istio-proxy"
Oct 30 01:44:22.459659 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.459614    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55134196-9552-406d-931e-b95123dcbd94" containerName="istio-proxy"
Oct 30 01:44:22.459659 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.459627    4864 cpu_manager.go:414] "containerMap: removing container" podUID="28a7413b-beb4-4b9f-bc76-7b1c5497ce1d" containerName="main"
Oct 30 01:44:22.459659 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.459633    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a7413b-beb4-4b9f-bc76-7b1c5497ce1d" containerName="main"
Oct 30 01:44:22.459769 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.459760    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="55134196-9552-406d-931e-b95123dcbd94" containerName="istio-proxy"
Oct 30 01:44:22.459801 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.459770    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a7413b-beb4-4b9f-bc76-7b1c5497ce1d" containerName="main"
Oct 30 01:44:22.459801 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.459782    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="55134196-9552-406d-931e-b95123dcbd94" containerName="istio-proxy"
Oct 30 01:44:22.464629 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.464608    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.496157 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.496127    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"]
Oct 30 01:44:22.620046 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.620010    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.620230 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.620093    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.620230 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.620179    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc75808d-fb11-4820-939a-18037b0c2791-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.620303 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.620231    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.620303 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.620290    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj9zf\" (UniqueName: \"kubernetes.io/projected/bc75808d-fb11-4820-939a-18037b0c2791-kube-api-access-zj9zf\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.620377 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.620321    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.620435 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.620412    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.620500 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.620488    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc75808d-fb11-4820-939a-18037b0c2791-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.620571 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.620553    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc75808d-fb11-4820-939a-18037b0c2791-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.721695 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.721611    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc75808d-fb11-4820-939a-18037b0c2791-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.721695 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.721668    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc75808d-fb11-4820-939a-18037b0c2791-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.721891 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.721830    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.721891 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.721879    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.721975 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.721908    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc75808d-fb11-4820-939a-18037b0c2791-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.721975 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.721950    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.722057 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.722006    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj9zf\" (UniqueName: \"kubernetes.io/projected/bc75808d-fb11-4820-939a-18037b0c2791-kube-api-access-zj9zf\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.722165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.722051    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.722165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.722120    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.722338 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.722309    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.722541 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.722342    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.722632 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.722365    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.722678 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.722659    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.722723 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.722692    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc75808d-fb11-4820-939a-18037b0c2791-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.724357 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.724335    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.724442 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.724366    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc75808d-fb11-4820-939a-18037b0c2791-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.746391 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.746357    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc75808d-fb11-4820-939a-18037b0c2791-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.746578 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.746556    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj9zf\" (UniqueName: \"kubernetes.io/projected/bc75808d-fb11-4820-939a-18037b0c2791-kube-api-access-zj9zf\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6hcmx\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:22.779629 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:22.779601    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:23.101194 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:23.101167    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"]
Oct 30 01:44:23.101774 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:44:23.101745    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc75808d_fb11_4820_939a_18037b0c2791.slice/crio-506e3770a45f1fa060a26cae8ec1684d0101c981a9cc1ad1c0d51fa556dd142f WatchSource:0}: Error finding container 506e3770a45f1fa060a26cae8ec1684d0101c981a9cc1ad1c0d51fa556dd142f: Status 404 returned error can't find the container with id 506e3770a45f1fa060a26cae8ec1684d0101c981a9cc1ad1c0d51fa556dd142f
Oct 30 01:44:23.104740 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:23.104704    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:44:23.104825 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:23.104773    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:44:23.104825 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:23.104804    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:44:23.711299 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:23.711256    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx" event={"ID":"bc75808d-fb11-4820-939a-18037b0c2791","Type":"ContainerStarted","Data":"49b43214c8922cf236068aa83328006e70e5f4b5d6c9a6d70a55803393cf0e2f"}
Oct 30 01:44:23.711813 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:23.711328    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx" event={"ID":"bc75808d-fb11-4820-939a-18037b0c2791","Type":"ContainerStarted","Data":"506e3770a45f1fa060a26cae8ec1684d0101c981a9cc1ad1c0d51fa556dd142f"}
Oct 30 01:44:23.765315 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:23.765252    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx" podStartSLOduration=1.7651401519999999 podStartE2EDuration="1.765140152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:44:23.763962665 +0000 UTC m=+3776.102064585" watchObservedRunningTime="2026-10-30 01:44:23.765140152 +0000 UTC m=+3776.103242063"
Oct 30 01:44:23.781216 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:23.781180    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:24.783428 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:24.783388    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6hcmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 01:44:24.783955 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:24.783445    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx" podUID="bc75808d-fb11-4820-939a-18037b0c2791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 01:44:25.781464 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:25.781424    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6hcmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 01:44:25.781647 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:25.781488    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx" podUID="bc75808d-fb11-4820-939a-18037b0c2791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 01:44:26.781470 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:26.781431    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6hcmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 01:44:26.781874 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:26.781500    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx" podUID="bc75808d-fb11-4820-939a-18037b0c2791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 01:44:27.781099 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:27.780441    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6hcmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 01:44:27.781099 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:27.780506    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx" podUID="bc75808d-fb11-4820-939a-18037b0c2791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 01:44:28.781485 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:28.781445    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6hcmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 01:44:28.781911 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:28.781500    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx" podUID="bc75808d-fb11-4820-939a-18037b0c2791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 01:44:29.780782 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:29.780742    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6hcmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 01:44:29.780949 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:29.780812    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx" podUID="bc75808d-fb11-4820-939a-18037b0c2791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 01:44:30.780374 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:30.780339    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6hcmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 01:44:30.780806 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:30.780399    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx" podUID="bc75808d-fb11-4820-939a-18037b0c2791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 01:44:31.785221 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:31.785188    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:31.794547 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:31.794523    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:31.794724 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:31.794593    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:44:49.041193 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:44:49.041157    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:44:53.017433 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:53.017396    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"]
Oct 30 01:44:53.017910 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:53.017636    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb" podUID="09577058-17aa-411b-bf71-092730f470f2" containerName="istio-proxy" containerID="cri-o://074ba6540b2eea261321bd974b7db80c6ea682f640127e77a5e6dfa440002c2f" gracePeriod=30
Oct 30 01:44:58.307109 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.307061    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:44:58.445122 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.442846    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-istio-data\") pod \"09577058-17aa-411b-bf71-092730f470f2\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") "
Oct 30 01:44:58.445122 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.442937    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09577058-17aa-411b-bf71-092730f470f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09577058-17aa-411b-bf71-092730f470f2-istiod-ca-cert\") pod \"09577058-17aa-411b-bf71-092730f470f2\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") "
Oct 30 01:44:58.445122 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.443004    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09577058-17aa-411b-bf71-092730f470f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09577058-17aa-411b-bf71-092730f470f2-istio-podinfo\") pod \"09577058-17aa-411b-bf71-092730f470f2\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") "
Oct 30 01:44:58.445122 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.443121    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-workload-socket\") pod \"09577058-17aa-411b-bf71-092730f470f2\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") "
Oct 30 01:44:58.445122 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.443179    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-istio-envoy\") pod \"09577058-17aa-411b-bf71-092730f470f2\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") "
Oct 30 01:44:58.445122 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.443214    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09577058-17aa-411b-bf71-092730f470f2-kube-api-access-6nh58\" (UniqueName: \"kubernetes.io/projected/09577058-17aa-411b-bf71-092730f470f2-kube-api-access-6nh58\") pod \"09577058-17aa-411b-bf71-092730f470f2\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") "
Oct 30 01:44:58.445122 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.443274    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-workload-certs\") pod \"09577058-17aa-411b-bf71-092730f470f2\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") "
Oct 30 01:44:58.445122 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.443375    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-credential-socket\") pod \"09577058-17aa-411b-bf71-092730f470f2\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") "
Oct 30 01:44:58.445122 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.443462    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09577058-17aa-411b-bf71-092730f470f2-istio-token\" (UniqueName: \"kubernetes.io/projected/09577058-17aa-411b-bf71-092730f470f2-istio-token\") pod \"09577058-17aa-411b-bf71-092730f470f2\" (UID: \"09577058-17aa-411b-bf71-092730f470f2\") "
Oct 30 01:44:58.445700 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.445337    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-workload-socket" pod "09577058-17aa-411b-bf71-092730f470f2" (UID: "09577058-17aa-411b-bf71-092730f470f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:44:58.445700 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.445549    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-istio-data" pod "09577058-17aa-411b-bf71-092730f470f2" (UID: "09577058-17aa-411b-bf71-092730f470f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:44:58.446141 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.445842    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09577058-17aa-411b-bf71-092730f470f2-istiod-ca-cert" pod "09577058-17aa-411b-bf71-092730f470f2" (UID: "09577058-17aa-411b-bf71-092730f470f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:44:58.455091 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.447649    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-credential-socket" pod "09577058-17aa-411b-bf71-092730f470f2" (UID: "09577058-17aa-411b-bf71-092730f470f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:44:58.455091 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.447789    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-workload-certs" pod "09577058-17aa-411b-bf71-092730f470f2" (UID: "09577058-17aa-411b-bf71-092730f470f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:44:58.455091 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.448180    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09577058-17aa-411b-bf71-092730f470f2-istio-token" pod "09577058-17aa-411b-bf71-092730f470f2" (UID: "09577058-17aa-411b-bf71-092730f470f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:44:58.456320 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.456284    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09577058-17aa-411b-bf71-092730f470f2-istio-podinfo" pod "09577058-17aa-411b-bf71-092730f470f2" (UID: "09577058-17aa-411b-bf71-092730f470f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:44:58.456543 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.456514    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-istio-envoy" pod "09577058-17aa-411b-bf71-092730f470f2" (UID: "09577058-17aa-411b-bf71-092730f470f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:44:58.456834 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.456818    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09577058-17aa-411b-bf71-092730f470f2-kube-api-access-6nh58" pod "09577058-17aa-411b-bf71-092730f470f2" (UID: "09577058-17aa-411b-bf71-092730f470f2"). InnerVolumeSpecName "kube-api-access-6nh58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:44:58.546492 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.546415    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:44:58.546492 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.546443    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09577058-17aa-411b-bf71-092730f470f2-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:44:58.546492 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.546453    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09577058-17aa-411b-bf71-092730f470f2-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:44:58.546492 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.546461    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:44:58.546492 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.546470    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:44:58.546492 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.546479    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nh58\" (UniqueName: \"kubernetes.io/projected/09577058-17aa-411b-bf71-092730f470f2-kube-api-access-6nh58\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:44:58.546492 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.546488    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:44:58.546492 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.546496    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09577058-17aa-411b-bf71-092730f470f2-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:44:58.546899 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.546505    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09577058-17aa-411b-bf71-092730f470f2-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:44:58.990759 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.990726    4864 generic.go:356] "Generic (PLEG): container finished" podID="09577058-17aa-411b-bf71-092730f470f2" containerID="074ba6540b2eea261321bd974b7db80c6ea682f640127e77a5e6dfa440002c2f" exitCode=0
Oct 30 01:44:58.990955 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.990801    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"
Oct 30 01:44:58.990955 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.990813    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb" event={"ID":"09577058-17aa-411b-bf71-092730f470f2","Type":"ContainerDied","Data":"074ba6540b2eea261321bd974b7db80c6ea682f640127e77a5e6dfa440002c2f"}
Oct 30 01:44:58.990955 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.990874    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb" event={"ID":"09577058-17aa-411b-bf71-092730f470f2","Type":"ContainerDied","Data":"ea8359d16dcb192b4ce08c9587cc185880a20296b33cbfa0ecfb88b33c020162"}
Oct 30 01:44:58.990955 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:58.990900    4864 scope.go:122] "RemoveContainer" containerID="074ba6540b2eea261321bd974b7db80c6ea682f640127e77a5e6dfa440002c2f"
Oct 30 01:44:59.003668 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:59.003649    4864 scope.go:122] "RemoveContainer" containerID="56af3c0c24f8558222890a96e15ced297db12fe990addeaf2469802fc725f416"
Oct 30 01:44:59.013670 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:59.013646    4864 scope.go:122] "RemoveContainer" containerID="074ba6540b2eea261321bd974b7db80c6ea682f640127e77a5e6dfa440002c2f"
Oct 30 01:44:59.013965 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:44:59.013940    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"074ba6540b2eea261321bd974b7db80c6ea682f640127e77a5e6dfa440002c2f\": container with ID starting with 074ba6540b2eea261321bd974b7db80c6ea682f640127e77a5e6dfa440002c2f not found: ID does not exist" containerID="074ba6540b2eea261321bd974b7db80c6ea682f640127e77a5e6dfa440002c2f"
Oct 30 01:44:59.014127 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:59.013970    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"074ba6540b2eea261321bd974b7db80c6ea682f640127e77a5e6dfa440002c2f"} err="failed to get container status \"074ba6540b2eea261321bd974b7db80c6ea682f640127e77a5e6dfa440002c2f\": rpc error: code = NotFound desc = could not find container \"074ba6540b2eea261321bd974b7db80c6ea682f640127e77a5e6dfa440002c2f\": container with ID starting with 074ba6540b2eea261321bd974b7db80c6ea682f640127e77a5e6dfa440002c2f not found: ID does not exist"
Oct 30 01:44:59.014127 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:59.013989    4864 scope.go:122] "RemoveContainer" containerID="56af3c0c24f8558222890a96e15ced297db12fe990addeaf2469802fc725f416"
Oct 30 01:44:59.014319 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:44:59.014300    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56af3c0c24f8558222890a96e15ced297db12fe990addeaf2469802fc725f416\": container with ID starting with 56af3c0c24f8558222890a96e15ced297db12fe990addeaf2469802fc725f416 not found: ID does not exist" containerID="56af3c0c24f8558222890a96e15ced297db12fe990addeaf2469802fc725f416"
Oct 30 01:44:59.014382 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:59.014324    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56af3c0c24f8558222890a96e15ced297db12fe990addeaf2469802fc725f416"} err="failed to get container status \"56af3c0c24f8558222890a96e15ced297db12fe990addeaf2469802fc725f416\": rpc error: code = NotFound desc = could not find container \"56af3c0c24f8558222890a96e15ced297db12fe990addeaf2469802fc725f416\": container with ID starting with 56af3c0c24f8558222890a96e15ced297db12fe990addeaf2469802fc725f416 not found: ID does not exist"
Oct 30 01:44:59.049463 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:59.049432    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"]
Oct 30 01:44:59.059790 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:59.059756    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b29d4a-openshift-default-b64cb9687-vqqfb"]
Oct 30 01:44:59.795188 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:44:59.795155    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09577058-17aa-411b-bf71-092730f470f2" path="/var/lib/kubelet/pods/09577058-17aa-411b-bf71-092730f470f2/volumes"
Oct 30 01:45:00.256581 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.256542    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc"]
Oct 30 01:45:00.257553 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.257531    4864 cpu_manager.go:414] "containerMap: removing container" podUID="09577058-17aa-411b-bf71-092730f470f2" containerName="istio-proxy"
Oct 30 01:45:00.257635 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.257554    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09577058-17aa-411b-bf71-092730f470f2" containerName="istio-proxy"
Oct 30 01:45:00.257635 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.257584    4864 cpu_manager.go:414] "containerMap: removing container" podUID="09577058-17aa-411b-bf71-092730f470f2" containerName="istio-proxy"
Oct 30 01:45:00.257635 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.257589    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09577058-17aa-411b-bf71-092730f470f2" containerName="istio-proxy"
Oct 30 01:45:00.257635 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.257606    4864 cpu_manager.go:414] "containerMap: removing container" podUID="55134196-9552-406d-931e-b95123dcbd94" containerName="istio-proxy"
Oct 30 01:45:00.257635 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.257623    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55134196-9552-406d-931e-b95123dcbd94" containerName="istio-proxy"
Oct 30 01:45:00.257901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.257739    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="09577058-17aa-411b-bf71-092730f470f2" containerName="istio-proxy"
Oct 30 01:45:00.264651 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.264627    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc"
Oct 30 01:45:00.269394 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.269366    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll"]
Oct 30 01:45:00.270264 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.270241    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="09577058-17aa-411b-bf71-092730f470f2" containerName="istio-proxy"
Oct 30 01:45:00.273823 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.273801    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll"
Oct 30 01:45:00.296359 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.296330    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc"]
Oct 30 01:45:00.307166 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.307137    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll"]
Oct 30 01:45:00.309115 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.309091    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq"]
Oct 30 01:45:00.314704 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.314683    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp"]
Oct 30 01:45:00.314820 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.314800    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq"
Oct 30 01:45:00.320342 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.320218    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp"
Oct 30 01:45:00.342968 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.342927    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq"]
Oct 30 01:45:00.348756 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.348733    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz"]
Oct 30 01:45:00.354353 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.354317    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp"]
Oct 30 01:45:00.354492 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.354452    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz"
Oct 30 01:45:00.371493 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.371435    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18da34b-a9cb-4c2b-b7a3-b7921ba4425e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc\" (UID: \"a18da34b-a9cb-4c2b-b7a3-b7921ba4425e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc"
Oct 30 01:45:00.372033 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.371924    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m748d\" (UniqueName: \"kubernetes.io/projected/a18da34b-a9cb-4c2b-b7a3-b7921ba4425e-kube-api-access-m748d\") pod \"maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc\" (UID: \"a18da34b-a9cb-4c2b-b7a3-b7921ba4425e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc"
Oct 30 01:45:00.372375 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.372180    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/241af13e-1e3a-466a-88b4-111de262ca2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll\" (UID: \"241af13e-1e3a-466a-88b4-111de262ca2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll"
Oct 30 01:45:00.372375 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.372232    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scxp4\" (UniqueName: \"kubernetes.io/projected/241af13e-1e3a-466a-88b4-111de262ca2a-kube-api-access-scxp4\") pod \"maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll\" (UID: \"241af13e-1e3a-466a-88b4-111de262ca2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll"
Oct 30 01:45:00.375677 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.375157    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz"]
Oct 30 01:45:00.381627 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.381601    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:45:00.390792 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.390758    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7"]
Oct 30 01:45:00.397505 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.397484    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7"]
Oct 30 01:45:00.397624 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.397604    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7"
Oct 30 01:45:00.404899 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.404275    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7"
Oct 30 01:45:00.413659 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.412229    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7"]
Oct 30 01:45:00.418237 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.415601    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7"]
Oct 30 01:45:00.473669 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.473635    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/241af13e-1e3a-466a-88b4-111de262ca2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll\" (UID: \"241af13e-1e3a-466a-88b4-111de262ca2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll"
Oct 30 01:45:00.473885 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.473686    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scxp4\" (UniqueName: \"kubernetes.io/projected/241af13e-1e3a-466a-88b4-111de262ca2a-kube-api-access-scxp4\") pod \"maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll\" (UID: \"241af13e-1e3a-466a-88b4-111de262ca2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll"
Oct 30 01:45:00.473885 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.473842    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18da34b-a9cb-4c2b-b7a3-b7921ba4425e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc\" (UID: \"a18da34b-a9cb-4c2b-b7a3-b7921ba4425e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc"
Oct 30 01:45:00.474028 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.473994    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp\" (UID: \"946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp"
Oct 30 01:45:00.474100 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.474039    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6pdx\" (UniqueName: \"kubernetes.io/projected/ba90b336-84e0-41ce-a673-29892a67100e-kube-api-access-c6pdx\") pod \"maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz\" (UID: \"ba90b336-84e0-41ce-a673-29892a67100e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz"
Oct 30 01:45:00.474180 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.474152    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba90b336-84e0-41ce-a673-29892a67100e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz\" (UID: \"ba90b336-84e0-41ce-a673-29892a67100e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz"
Oct 30 01:45:00.474297 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.474278    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbxks\" (UniqueName: \"kubernetes.io/projected/946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e-kube-api-access-sbxks\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp\" (UID: \"946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp"
Oct 30 01:45:00.474368 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.474317    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m748d\" (UniqueName: \"kubernetes.io/projected/a18da34b-a9cb-4c2b-b7a3-b7921ba4425e-kube-api-access-m748d\") pod \"maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc\" (UID: \"a18da34b-a9cb-4c2b-b7a3-b7921ba4425e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc"
Oct 30 01:45:00.474368 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.474356    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwm5p\" (UniqueName: \"kubernetes.io/projected/ce4812b0-4108-4d51-98cf-607aab4eebc6-kube-api-access-rwm5p\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq\" (UID: \"ce4812b0-4108-4d51-98cf-607aab4eebc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq"
Oct 30 01:45:00.474475 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.474388    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce4812b0-4108-4d51-98cf-607aab4eebc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq\" (UID: \"ce4812b0-4108-4d51-98cf-607aab4eebc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq"
Oct 30 01:45:00.474475 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.474396    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/241af13e-1e3a-466a-88b4-111de262ca2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll\" (UID: \"241af13e-1e3a-466a-88b4-111de262ca2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll"
Oct 30 01:45:00.474709 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.474686    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18da34b-a9cb-4c2b-b7a3-b7921ba4425e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc\" (UID: \"a18da34b-a9cb-4c2b-b7a3-b7921ba4425e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc"
Oct 30 01:45:00.506709 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.506635    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scxp4\" (UniqueName: \"kubernetes.io/projected/241af13e-1e3a-466a-88b4-111de262ca2a-kube-api-access-scxp4\") pod \"maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll\" (UID: \"241af13e-1e3a-466a-88b4-111de262ca2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll"
Oct 30 01:45:00.508403 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.508381    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m748d\" (UniqueName: \"kubernetes.io/projected/a18da34b-a9cb-4c2b-b7a3-b7921ba4425e-kube-api-access-m748d\") pod \"maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc\" (UID: \"a18da34b-a9cb-4c2b-b7a3-b7921ba4425e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc"
Oct 30 01:45:00.575971 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.575927    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zcmn\" (UniqueName: \"kubernetes.io/projected/e9245410-0ba2-4d32-abd2-cfa9e692fd78-kube-api-access-2zcmn\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7\" (UID: \"e9245410-0ba2-4d32-abd2-cfa9e692fd78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7"
Oct 30 01:45:00.576176 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.576015    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9245410-0ba2-4d32-abd2-cfa9e692fd78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7\" (UID: \"e9245410-0ba2-4d32-abd2-cfa9e692fd78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7"
Oct 30 01:45:00.576364 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.576335    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp\" (UID: \"946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp"
Oct 30 01:45:00.576428 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.576395    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6pdx\" (UniqueName: \"kubernetes.io/projected/ba90b336-84e0-41ce-a673-29892a67100e-kube-api-access-c6pdx\") pod \"maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz\" (UID: \"ba90b336-84e0-41ce-a673-29892a67100e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz"
Oct 30 01:45:00.576471 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.576433    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba90b336-84e0-41ce-a673-29892a67100e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz\" (UID: \"ba90b336-84e0-41ce-a673-29892a67100e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz"
Oct 30 01:45:00.576471 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.576462    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfpxt\" (UniqueName: \"kubernetes.io/projected/d9774ce4-b064-4823-a4dc-e915ff4820f0-kube-api-access-gfpxt\") pod \"maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7\" (UID: \"d9774ce4-b064-4823-a4dc-e915ff4820f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7"
Oct 30 01:45:00.576559 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.576492    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbxks\" (UniqueName: \"kubernetes.io/projected/946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e-kube-api-access-sbxks\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp\" (UID: \"946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp"
Oct 30 01:45:00.576559 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.576522    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwm5p\" (UniqueName: \"kubernetes.io/projected/ce4812b0-4108-4d51-98cf-607aab4eebc6-kube-api-access-rwm5p\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq\" (UID: \"ce4812b0-4108-4d51-98cf-607aab4eebc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq"
Oct 30 01:45:00.576559 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.576545    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce4812b0-4108-4d51-98cf-607aab4eebc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq\" (UID: \"ce4812b0-4108-4d51-98cf-607aab4eebc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq"
Oct 30 01:45:00.576726 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.576600    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9774ce4-b064-4823-a4dc-e915ff4820f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7\" (UID: \"d9774ce4-b064-4823-a4dc-e915ff4820f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7"
Oct 30 01:45:00.577170 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.577147    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp\" (UID: \"946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp"
Oct 30 01:45:00.577281 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.577210    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce4812b0-4108-4d51-98cf-607aab4eebc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq\" (UID: \"ce4812b0-4108-4d51-98cf-607aab4eebc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq"
Oct 30 01:45:00.577281 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.577250    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba90b336-84e0-41ce-a673-29892a67100e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz\" (UID: \"ba90b336-84e0-41ce-a673-29892a67100e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz"
Oct 30 01:45:00.586844 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.586818    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc"
Oct 30 01:45:00.598557 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.598527    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6pdx\" (UniqueName: \"kubernetes.io/projected/ba90b336-84e0-41ce-a673-29892a67100e-kube-api-access-c6pdx\") pod \"maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz\" (UID: \"ba90b336-84e0-41ce-a673-29892a67100e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz"
Oct 30 01:45:00.598644 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.598571    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbxks\" (UniqueName: \"kubernetes.io/projected/946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e-kube-api-access-sbxks\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp\" (UID: \"946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp"
Oct 30 01:45:00.598934 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.598916    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwm5p\" (UniqueName: \"kubernetes.io/projected/ce4812b0-4108-4d51-98cf-607aab4eebc6-kube-api-access-rwm5p\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq\" (UID: \"ce4812b0-4108-4d51-98cf-607aab4eebc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq"
Oct 30 01:45:00.604839 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.604812    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll"
Oct 30 01:45:00.634479 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.634439    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq"
Oct 30 01:45:00.658160 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.655331    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp"
Oct 30 01:45:00.695840 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.691709    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfpxt\" (UniqueName: \"kubernetes.io/projected/d9774ce4-b064-4823-a4dc-e915ff4820f0-kube-api-access-gfpxt\") pod \"maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7\" (UID: \"d9774ce4-b064-4823-a4dc-e915ff4820f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7"
Oct 30 01:45:00.695840 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.691800    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9774ce4-b064-4823-a4dc-e915ff4820f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7\" (UID: \"d9774ce4-b064-4823-a4dc-e915ff4820f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7"
Oct 30 01:45:00.695840 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.691846    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zcmn\" (UniqueName: \"kubernetes.io/projected/e9245410-0ba2-4d32-abd2-cfa9e692fd78-kube-api-access-2zcmn\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7\" (UID: \"e9245410-0ba2-4d32-abd2-cfa9e692fd78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7"
Oct 30 01:45:00.695840 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.691920    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9245410-0ba2-4d32-abd2-cfa9e692fd78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7\" (UID: \"e9245410-0ba2-4d32-abd2-cfa9e692fd78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7"
Oct 30 01:45:00.695840 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.693570    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz"
Oct 30 01:45:00.695840 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.695327    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9774ce4-b064-4823-a4dc-e915ff4820f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7\" (UID: \"d9774ce4-b064-4823-a4dc-e915ff4820f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7"
Oct 30 01:45:00.700890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.700862    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9245410-0ba2-4d32-abd2-cfa9e692fd78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7\" (UID: \"e9245410-0ba2-4d32-abd2-cfa9e692fd78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7"
Oct 30 01:45:00.721540 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.721510    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zcmn\" (UniqueName: \"kubernetes.io/projected/e9245410-0ba2-4d32-abd2-cfa9e692fd78-kube-api-access-2zcmn\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7\" (UID: \"e9245410-0ba2-4d32-abd2-cfa9e692fd78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7"
Oct 30 01:45:00.724392 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.724364    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfpxt\" (UniqueName: \"kubernetes.io/projected/d9774ce4-b064-4823-a4dc-e915ff4820f0-kube-api-access-gfpxt\") pod \"maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7\" (UID: \"d9774ce4-b064-4823-a4dc-e915ff4820f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7"
Oct 30 01:45:00.731471 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.731421    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7"
Oct 30 01:45:00.915423 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:00.913315    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc"]
Oct 30 01:45:00.915423 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:45:00.914912    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18da34b_a9cb_4c2b_b7a3_b7921ba4425e.slice/crio-5dd045246988bb04aee36ceffa04fbe91ae2e74be6b03223c49d88893836c84f WatchSource:0}: Error finding container 5dd045246988bb04aee36ceffa04fbe91ae2e74be6b03223c49d88893836c84f: Status 404 returned error can't find the container with id 5dd045246988bb04aee36ceffa04fbe91ae2e74be6b03223c49d88893836c84f
Oct 30 01:45:01.015159 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:01.014819    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc" event={"ID":"a18da34b-a9cb-4c2b-b7a3-b7921ba4425e","Type":"ContainerStarted","Data":"5dd045246988bb04aee36ceffa04fbe91ae2e74be6b03223c49d88893836c84f"}
Oct 30 01:45:01.022171 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:01.022144    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7"
Oct 30 01:45:01.080351 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:01.080322    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7"]
Oct 30 01:45:01.127616 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:01.127588    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll"]
Oct 30 01:45:01.159139 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:45:01.159065    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9774ce4_b064_4823_a4dc_e915ff4820f0.slice/crio-e2c9297f27797650d58fab5cb0906a7673c0f93d31883bda8e69017c3b3b4005 WatchSource:0}: Error finding container e2c9297f27797650d58fab5cb0906a7673c0f93d31883bda8e69017c3b3b4005: Status 404 returned error can't find the container with id e2c9297f27797650d58fab5cb0906a7673c0f93d31883bda8e69017c3b3b4005
Oct 30 01:45:01.160557 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:45:01.160519    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241af13e_1e3a_466a_88b4_111de262ca2a.slice/crio-b6115e06d6d0b866464776b720f484c87c92c6ba2a25280cbc28b475ba5ea90e WatchSource:0}: Error finding container b6115e06d6d0b866464776b720f484c87c92c6ba2a25280cbc28b475ba5ea90e: Status 404 returned error can't find the container with id b6115e06d6d0b866464776b720f484c87c92c6ba2a25280cbc28b475ba5ea90e
Oct 30 01:45:01.212833 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:01.212806    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq"]
Oct 30 01:45:01.212946 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:45:01.212853    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce4812b0_4108_4d51_98cf_607aab4eebc6.slice/crio-232b6bee313f406f1e21f6d714f8713bbdbc7e800a54a990b5fd92e06fe633ee WatchSource:0}: Error finding container 232b6bee313f406f1e21f6d714f8713bbdbc7e800a54a990b5fd92e06fe633ee: Status 404 returned error can't find the container with id 232b6bee313f406f1e21f6d714f8713bbdbc7e800a54a990b5fd92e06fe633ee
Oct 30 01:45:01.230134 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:01.226712    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp"]
Oct 30 01:45:01.236897 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:01.236871    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz"]
Oct 30 01:45:01.237039 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:45:01.237012    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946d9b3b_d10a_4ccd_ae80_ffe5c421ff2e.slice/crio-126128261ac332f9e325a1b4334f7beb468b134af9b55c5147bd972ab95f9ec9 WatchSource:0}: Error finding container 126128261ac332f9e325a1b4334f7beb468b134af9b55c5147bd972ab95f9ec9: Status 404 returned error can't find the container with id 126128261ac332f9e325a1b4334f7beb468b134af9b55c5147bd972ab95f9ec9
Oct 30 01:45:01.340886 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:01.340861    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7"]
Oct 30 01:45:01.360974 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:45:01.360934    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9245410_0ba2_4d32_abd2_cfa9e692fd78.slice/crio-312c995b681193ee30e36214b9d408bcfc1b0ea262a7eeafe4b983a7be897f58 WatchSource:0}: Error finding container 312c995b681193ee30e36214b9d408bcfc1b0ea262a7eeafe4b983a7be897f58: Status 404 returned error can't find the container with id 312c995b681193ee30e36214b9d408bcfc1b0ea262a7eeafe4b983a7be897f58
Oct 30 01:45:01.626595 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:01.626520    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd"]
Oct 30 01:45:01.652619 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:01.652587    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g"]
Oct 30 01:45:02.024163 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:02.024124    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc" event={"ID":"a18da34b-a9cb-4c2b-b7a3-b7921ba4425e","Type":"ContainerStarted","Data":"71fd454cce39b4c14dc668821107d6a49521908a7e75ba5de9fc0acfe46e4700"}
Oct 30 01:45:02.026024 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:02.025995    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7" event={"ID":"d9774ce4-b064-4823-a4dc-e915ff4820f0","Type":"ContainerStarted","Data":"c4da91b15087265cef1d080b2015b189dfa2de3cba258dcdf48d235c724fbf6f"}
Oct 30 01:45:02.026161 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:02.026042    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7" event={"ID":"d9774ce4-b064-4823-a4dc-e915ff4820f0","Type":"ContainerStarted","Data":"e2c9297f27797650d58fab5cb0906a7673c0f93d31883bda8e69017c3b3b4005"}
Oct 30 01:45:02.027220 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:02.027196    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7" event={"ID":"e9245410-0ba2-4d32-abd2-cfa9e692fd78","Type":"ContainerStarted","Data":"312c995b681193ee30e36214b9d408bcfc1b0ea262a7eeafe4b983a7be897f58"}
Oct 30 01:45:02.028812 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:02.028789    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll" event={"ID":"241af13e-1e3a-466a-88b4-111de262ca2a","Type":"ContainerStarted","Data":"958f97f4b6b4b2e6e5b1658db0517fa1ca0f6103c856a421d151e291c6e19c66"}
Oct 30 01:45:02.028935 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:02.028829    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll" event={"ID":"241af13e-1e3a-466a-88b4-111de262ca2a","Type":"ContainerStarted","Data":"b6115e06d6d0b866464776b720f484c87c92c6ba2a25280cbc28b475ba5ea90e"}
Oct 30 01:45:02.030582 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:02.030558    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz" event={"ID":"ba90b336-84e0-41ce-a673-29892a67100e","Type":"ContainerStarted","Data":"ac5c2e176266c666b85046c5cfb5e76d435f85a3b6491fe63ddcaae448c15949"}
Oct 30 01:45:02.030685 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:02.030602    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz" event={"ID":"ba90b336-84e0-41ce-a673-29892a67100e","Type":"ContainerStarted","Data":"3f983522c904e08d590dbc5660b039e2f210a2f13a00f8cbf5403a830d3ba031"}
Oct 30 01:45:02.032150 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:02.032122    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp" event={"ID":"946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e","Type":"ContainerStarted","Data":"aa32c2aab15d54c0bd4e5c997ed736109dd59155e184c42edf3459fa3b05e264"}
Oct 30 01:45:02.032238 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:02.032170    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp" event={"ID":"946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e","Type":"ContainerStarted","Data":"126128261ac332f9e325a1b4334f7beb468b134af9b55c5147bd972ab95f9ec9"}
Oct 30 01:45:02.033784 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:02.033757    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq" event={"ID":"ce4812b0-4108-4d51-98cf-607aab4eebc6","Type":"ContainerStarted","Data":"3923af831fa9f73c0e6188cbf6f77304a7ab0c33c4c9d481d523509ca35c9e8f"}
Oct 30 01:45:02.033874 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:02.033804    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq" event={"ID":"ce4812b0-4108-4d51-98cf-607aab4eebc6","Type":"ContainerStarted","Data":"232b6bee313f406f1e21f6d714f8713bbdbc7e800a54a990b5fd92e06fe633ee"}
Oct 30 01:45:02.062747 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:02.062691    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc" podStartSLOduration=2.062673656 podStartE2EDuration="2.062673656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:02.062313661 +0000 UTC m=+3814.400415575" watchObservedRunningTime="2026-10-30 01:45:02.062673656 +0000 UTC m=+3814.400775569"
Oct 30 01:45:02.099105 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:02.099040    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll" podStartSLOduration=2.099021907 podStartE2EDuration="2.099021907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:02.09785425 +0000 UTC m=+3814.435956165" watchObservedRunningTime="2026-10-30 01:45:02.099021907 +0000 UTC m=+3814.437123817"
Oct 30 01:45:02.136130 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:02.136026    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz" podStartSLOduration=2.136010896 podStartE2EDuration="2.136010896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:02.134146629 +0000 UTC m=+3814.472248538" watchObservedRunningTime="2026-10-30 01:45:02.136010896 +0000 UTC m=+3814.474112846"
Oct 30 01:45:02.174201 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:02.174153    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7" podStartSLOduration=2.174138842 podStartE2EDuration="2.174138842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:02.169920146 +0000 UTC m=+3814.508022071" watchObservedRunningTime="2026-10-30 01:45:02.174138842 +0000 UTC m=+3814.512240753"
Oct 30 01:45:02.274126 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:02.273132    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq" podStartSLOduration=2.273111391 podStartE2EDuration="2.273111391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:02.209611414 +0000 UTC m=+3814.547713325" watchObservedRunningTime="2026-10-30 01:45:02.273111391 +0000 UTC m=+3814.611213303"
Oct 30 01:45:02.274335 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:02.274284    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp" podStartSLOduration=2.274272401 podStartE2EDuration="2.274272401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:02.26562986 +0000 UTC m=+3814.603731771" watchObservedRunningTime="2026-10-30 01:45:02.274272401 +0000 UTC m=+3814.612374314"
Oct 30 01:45:02.679848 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:02.679797    4864 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba90b336_84e0_41ce_a673_29892a67100e.slice/crio-ac5c2e176266c666b85046c5cfb5e76d435f85a3b6491fe63ddcaae448c15949.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba90b336_84e0_41ce_a673_29892a67100e.slice/crio-conmon-ac5c2e176266c666b85046c5cfb5e76d435f85a3b6491fe63ddcaae448c15949.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 01:45:03.042323 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:03.042288    4864 generic.go:356] "Generic (PLEG): container finished" podID="a18da34b-a9cb-4c2b-b7a3-b7921ba4425e" containerID="71fd454cce39b4c14dc668821107d6a49521908a7e75ba5de9fc0acfe46e4700" exitCode=60
Oct 30 01:45:03.042733 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:03.042365    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc" event={"ID":"a18da34b-a9cb-4c2b-b7a3-b7921ba4425e","Type":"ContainerDied","Data":"71fd454cce39b4c14dc668821107d6a49521908a7e75ba5de9fc0acfe46e4700"}
Oct 30 01:45:03.042733 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:03.042689    4864 scope.go:122] "RemoveContainer" containerID="71fd454cce39b4c14dc668821107d6a49521908a7e75ba5de9fc0acfe46e4700"
Oct 30 01:45:03.044126 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:03.044067    4864 generic.go:356] "Generic (PLEG): container finished" podID="d9774ce4-b064-4823-a4dc-e915ff4820f0" containerID="c4da91b15087265cef1d080b2015b189dfa2de3cba258dcdf48d235c724fbf6f" exitCode=60
Oct 30 01:45:03.044282 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:03.044161    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7" event={"ID":"d9774ce4-b064-4823-a4dc-e915ff4820f0","Type":"ContainerDied","Data":"c4da91b15087265cef1d080b2015b189dfa2de3cba258dcdf48d235c724fbf6f"}
Oct 30 01:45:03.044438 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:03.044422    4864 scope.go:122] "RemoveContainer" containerID="c4da91b15087265cef1d080b2015b189dfa2de3cba258dcdf48d235c724fbf6f"
Oct 30 01:45:03.046388 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:03.046307    4864 generic.go:356] "Generic (PLEG): container finished" podID="e9245410-0ba2-4d32-abd2-cfa9e692fd78" containerID="b9bf412696b9d9467d650f96cb3f4edf3858ff636d53a79c50f7f9ddd042f4dd" exitCode=60
Oct 30 01:45:03.046388 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:03.046372    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7" event={"ID":"e9245410-0ba2-4d32-abd2-cfa9e692fd78","Type":"ContainerDied","Data":"b9bf412696b9d9467d650f96cb3f4edf3858ff636d53a79c50f7f9ddd042f4dd"}
Oct 30 01:45:03.046671 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:03.046652    4864 scope.go:122] "RemoveContainer" containerID="b9bf412696b9d9467d650f96cb3f4edf3858ff636d53a79c50f7f9ddd042f4dd"
Oct 30 01:45:03.048140 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:03.048119    4864 generic.go:356] "Generic (PLEG): container finished" podID="241af13e-1e3a-466a-88b4-111de262ca2a" containerID="958f97f4b6b4b2e6e5b1658db0517fa1ca0f6103c856a421d151e291c6e19c66" exitCode=60
Oct 30 01:45:03.048396 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:03.048227    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll" event={"ID":"241af13e-1e3a-466a-88b4-111de262ca2a","Type":"ContainerDied","Data":"958f97f4b6b4b2e6e5b1658db0517fa1ca0f6103c856a421d151e291c6e19c66"}
Oct 30 01:45:03.048512 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:03.048461    4864 scope.go:122] "RemoveContainer" containerID="958f97f4b6b4b2e6e5b1658db0517fa1ca0f6103c856a421d151e291c6e19c66"
Oct 30 01:45:03.049918 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:03.049894    4864 generic.go:356] "Generic (PLEG): container finished" podID="ba90b336-84e0-41ce-a673-29892a67100e" containerID="ac5c2e176266c666b85046c5cfb5e76d435f85a3b6491fe63ddcaae448c15949" exitCode=60
Oct 30 01:45:03.050042 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:03.049969    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz" event={"ID":"ba90b336-84e0-41ce-a673-29892a67100e","Type":"ContainerDied","Data":"ac5c2e176266c666b85046c5cfb5e76d435f85a3b6491fe63ddcaae448c15949"}
Oct 30 01:45:03.050201 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:03.050181    4864 scope.go:122] "RemoveContainer" containerID="ac5c2e176266c666b85046c5cfb5e76d435f85a3b6491fe63ddcaae448c15949"
Oct 30 01:45:03.059696 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:03.059645    4864 generic.go:356] "Generic (PLEG): container finished" podID="946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e" containerID="aa32c2aab15d54c0bd4e5c997ed736109dd59155e184c42edf3459fa3b05e264" exitCode=60
Oct 30 01:45:03.059864 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:03.059800    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp" event={"ID":"946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e","Type":"ContainerDied","Data":"aa32c2aab15d54c0bd4e5c997ed736109dd59155e184c42edf3459fa3b05e264"}
Oct 30 01:45:03.060022 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:03.060006    4864 scope.go:122] "RemoveContainer" containerID="aa32c2aab15d54c0bd4e5c997ed736109dd59155e184c42edf3459fa3b05e264"
Oct 30 01:45:03.073246 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:03.072990    4864 generic.go:356] "Generic (PLEG): container finished" podID="ce4812b0-4108-4d51-98cf-607aab4eebc6" containerID="3923af831fa9f73c0e6188cbf6f77304a7ab0c33c4c9d481d523509ca35c9e8f" exitCode=60
Oct 30 01:45:03.073246 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:03.073044    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq" event={"ID":"ce4812b0-4108-4d51-98cf-607aab4eebc6","Type":"ContainerDied","Data":"3923af831fa9f73c0e6188cbf6f77304a7ab0c33c4c9d481d523509ca35c9e8f"}
Oct 30 01:45:03.073485 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:03.073468    4864 scope.go:122] "RemoveContainer" containerID="3923af831fa9f73c0e6188cbf6f77304a7ab0c33c4c9d481d523509ca35c9e8f"
Oct 30 01:45:04.082525 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:04.082494    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc" event={"ID":"a18da34b-a9cb-4c2b-b7a3-b7921ba4425e","Type":"ContainerStarted","Data":"86780da58dca1f3094e735362b441ec88983aadf62146eb169a4913c2e7d5d76"}
Oct 30 01:45:04.084441 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:04.084417    4864 generic.go:356] "Generic (PLEG): container finished" podID="d9774ce4-b064-4823-a4dc-e915ff4820f0" containerID="48a87dc257cbc36b0587dbbe59b2f74c9017e369b1b0c1e80f5398d93df71543" exitCode=60
Oct 30 01:45:04.084555 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:04.084504    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7" event={"ID":"d9774ce4-b064-4823-a4dc-e915ff4820f0","Type":"ContainerDied","Data":"48a87dc257cbc36b0587dbbe59b2f74c9017e369b1b0c1e80f5398d93df71543"}
Oct 30 01:45:04.084608 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:04.084563    4864 scope.go:122] "RemoveContainer" containerID="c4da91b15087265cef1d080b2015b189dfa2de3cba258dcdf48d235c724fbf6f"
Oct 30 01:45:04.084795 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:04.084777    4864 scope.go:122] "RemoveContainer" containerID="48a87dc257cbc36b0587dbbe59b2f74c9017e369b1b0c1e80f5398d93df71543"
Oct 30 01:45:04.085064 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:04.085040    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7_odh-ai-gateway-infra(d9774ce4-b064-4823-a4dc-e915ff4820f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7" podUID="d9774ce4-b064-4823-a4dc-e915ff4820f0"
Oct 30 01:45:04.086642 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:04.086621    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7" event={"ID":"e9245410-0ba2-4d32-abd2-cfa9e692fd78","Type":"ContainerStarted","Data":"2ded57d0c40b0305f8caa0f2ac57bb5ff71a601d01eb9fd78fe0ed2672ec2ca3"}
Oct 30 01:45:04.088499 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:04.088477    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll" event={"ID":"241af13e-1e3a-466a-88b4-111de262ca2a","Type":"ContainerStarted","Data":"2594f78b2362490bd5c09b7a633add4898af8b5db13fe3150ef7b77791b14df4"}
Oct 30 01:45:04.090758 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:04.090734    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz" event={"ID":"ba90b336-84e0-41ce-a673-29892a67100e","Type":"ContainerStarted","Data":"81c7d2fdf5cedf38a8604be56a3e5ef620374c621107fd17ef7d9a7b111805b6"}
Oct 30 01:45:04.092844 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:04.092818    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp" event={"ID":"946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e","Type":"ContainerStarted","Data":"21ba6529d04c063fc43e82807868fdaef1ba4d9f127cab67fef125a03ce39e82"}
Oct 30 01:45:04.094804 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:04.094784    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq" event={"ID":"ce4812b0-4108-4d51-98cf-607aab4eebc6","Type":"ContainerStarted","Data":"07adc3b3bcaa902c423e5c9a07d2f93efe884ffc6c3f5d95d94ef0eb04cd8fe8"}
Oct 30 01:45:04.479770 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:04.479722    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7" podStartSLOduration=4.479707213 podStartE2EDuration="4.479707213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:04.382444658 +0000 UTC m=+3816.720546571" watchObservedRunningTime="2026-10-30 01:45:04.479707213 +0000 UTC m=+3816.817809160"
Oct 30 01:45:05.105469 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.105434    4864 generic.go:356] "Generic (PLEG): container finished" podID="946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e" containerID="21ba6529d04c063fc43e82807868fdaef1ba4d9f127cab67fef125a03ce39e82" exitCode=60
Oct 30 01:45:05.105977 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.105519    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp" event={"ID":"946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e","Type":"ContainerDied","Data":"21ba6529d04c063fc43e82807868fdaef1ba4d9f127cab67fef125a03ce39e82"}
Oct 30 01:45:05.105977 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.105693    4864 scope.go:122] "RemoveContainer" containerID="aa32c2aab15d54c0bd4e5c997ed736109dd59155e184c42edf3459fa3b05e264"
Oct 30 01:45:05.105977 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.105782    4864 scope.go:122] "RemoveContainer" containerID="21ba6529d04c063fc43e82807868fdaef1ba4d9f127cab67fef125a03ce39e82"
Oct 30 01:45:05.106190 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:05.106104    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp_odh-ai-gateway-infra(946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp" podUID="946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e"
Oct 30 01:45:05.107809 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.107787    4864 generic.go:356] "Generic (PLEG): container finished" podID="ce4812b0-4108-4d51-98cf-607aab4eebc6" containerID="07adc3b3bcaa902c423e5c9a07d2f93efe884ffc6c3f5d95d94ef0eb04cd8fe8" exitCode=60
Oct 30 01:45:05.107895 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.107864    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq" event={"ID":"ce4812b0-4108-4d51-98cf-607aab4eebc6","Type":"ContainerDied","Data":"07adc3b3bcaa902c423e5c9a07d2f93efe884ffc6c3f5d95d94ef0eb04cd8fe8"}
Oct 30 01:45:05.108115 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.108066    4864 scope.go:122] "RemoveContainer" containerID="07adc3b3bcaa902c423e5c9a07d2f93efe884ffc6c3f5d95d94ef0eb04cd8fe8"
Oct 30 01:45:05.108372 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:05.108348    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq_odh-ai-gateway-infra(ce4812b0-4108-4d51-98cf-607aab4eebc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq" podUID="ce4812b0-4108-4d51-98cf-607aab4eebc6"
Oct 30 01:45:05.109957 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.109937    4864 generic.go:356] "Generic (PLEG): container finished" podID="a18da34b-a9cb-4c2b-b7a3-b7921ba4425e" containerID="86780da58dca1f3094e735362b441ec88983aadf62146eb169a4913c2e7d5d76" exitCode=60
Oct 30 01:45:05.110064 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.109972    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc" event={"ID":"a18da34b-a9cb-4c2b-b7a3-b7921ba4425e","Type":"ContainerDied","Data":"86780da58dca1f3094e735362b441ec88983aadf62146eb169a4913c2e7d5d76"}
Oct 30 01:45:05.110284 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.110268    4864 scope.go:122] "RemoveContainer" containerID="86780da58dca1f3094e735362b441ec88983aadf62146eb169a4913c2e7d5d76"
Oct 30 01:45:05.110678 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:05.110543    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc_odh-ai-gateway-infra(a18da34b-a9cb-4c2b-b7a3-b7921ba4425e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc" podUID="a18da34b-a9cb-4c2b-b7a3-b7921ba4425e"
Oct 30 01:45:05.111996 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.111979    4864 scope.go:122] "RemoveContainer" containerID="48a87dc257cbc36b0587dbbe59b2f74c9017e369b1b0c1e80f5398d93df71543"
Oct 30 01:45:05.112276 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:05.112252    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7_odh-ai-gateway-infra(d9774ce4-b064-4823-a4dc-e915ff4820f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7" podUID="d9774ce4-b064-4823-a4dc-e915ff4820f0"
Oct 30 01:45:05.113790 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.113770    4864 generic.go:356] "Generic (PLEG): container finished" podID="e9245410-0ba2-4d32-abd2-cfa9e692fd78" containerID="2ded57d0c40b0305f8caa0f2ac57bb5ff71a601d01eb9fd78fe0ed2672ec2ca3" exitCode=60
Oct 30 01:45:05.113888 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.113833    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7" event={"ID":"e9245410-0ba2-4d32-abd2-cfa9e692fd78","Type":"ContainerDied","Data":"2ded57d0c40b0305f8caa0f2ac57bb5ff71a601d01eb9fd78fe0ed2672ec2ca3"}
Oct 30 01:45:05.114011 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.113997    4864 scope.go:122] "RemoveContainer" containerID="2ded57d0c40b0305f8caa0f2ac57bb5ff71a601d01eb9fd78fe0ed2672ec2ca3"
Oct 30 01:45:05.114281 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:05.114258    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7_odh-ai-gateway-infra(e9245410-0ba2-4d32-abd2-cfa9e692fd78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7" podUID="e9245410-0ba2-4d32-abd2-cfa9e692fd78"
Oct 30 01:45:05.118722 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.118662    4864 generic.go:356] "Generic (PLEG): container finished" podID="241af13e-1e3a-466a-88b4-111de262ca2a" containerID="2594f78b2362490bd5c09b7a633add4898af8b5db13fe3150ef7b77791b14df4" exitCode=60
Oct 30 01:45:05.118845 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.118798    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll" event={"ID":"241af13e-1e3a-466a-88b4-111de262ca2a","Type":"ContainerDied","Data":"2594f78b2362490bd5c09b7a633add4898af8b5db13fe3150ef7b77791b14df4"}
Oct 30 01:45:05.119265 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.119247    4864 scope.go:122] "RemoveContainer" containerID="2594f78b2362490bd5c09b7a633add4898af8b5db13fe3150ef7b77791b14df4"
Oct 30 01:45:05.120237 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:05.119539    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll_odh-ai-gateway-infra(241af13e-1e3a-466a-88b4-111de262ca2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll" podUID="241af13e-1e3a-466a-88b4-111de262ca2a"
Oct 30 01:45:05.123249 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.123222    4864 generic.go:356] "Generic (PLEG): container finished" podID="ba90b336-84e0-41ce-a673-29892a67100e" containerID="81c7d2fdf5cedf38a8604be56a3e5ef620374c621107fd17ef7d9a7b111805b6" exitCode=60
Oct 30 01:45:05.123424 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.123261    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz" event={"ID":"ba90b336-84e0-41ce-a673-29892a67100e","Type":"ContainerDied","Data":"81c7d2fdf5cedf38a8604be56a3e5ef620374c621107fd17ef7d9a7b111805b6"}
Oct 30 01:45:05.123424 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.123382    4864 scope.go:122] "RemoveContainer" containerID="3923af831fa9f73c0e6188cbf6f77304a7ab0c33c4c9d481d523509ca35c9e8f"
Oct 30 01:45:05.123573 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.123553    4864 scope.go:122] "RemoveContainer" containerID="81c7d2fdf5cedf38a8604be56a3e5ef620374c621107fd17ef7d9a7b111805b6"
Oct 30 01:45:05.123841 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:05.123822    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz_odh-ai-gateway-infra(ba90b336-84e0-41ce-a673-29892a67100e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz" podUID="ba90b336-84e0-41ce-a673-29892a67100e"
Oct 30 01:45:05.138030 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.138010    4864 scope.go:122] "RemoveContainer" containerID="71fd454cce39b4c14dc668821107d6a49521908a7e75ba5de9fc0acfe46e4700"
Oct 30 01:45:05.152504 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.152486    4864 scope.go:122] "RemoveContainer" containerID="b9bf412696b9d9467d650f96cb3f4edf3858ff636d53a79c50f7f9ddd042f4dd"
Oct 30 01:45:05.167896 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.167872    4864 scope.go:122] "RemoveContainer" containerID="958f97f4b6b4b2e6e5b1658db0517fa1ca0f6103c856a421d151e291c6e19c66"
Oct 30 01:45:05.181791 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:05.181766    4864 scope.go:122] "RemoveContainer" containerID="ac5c2e176266c666b85046c5cfb5e76d435f85a3b6491fe63ddcaae448c15949"
Oct 30 01:45:06.132506 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:06.132472    4864 scope.go:122] "RemoveContainer" containerID="86780da58dca1f3094e735362b441ec88983aadf62146eb169a4913c2e7d5d76"
Oct 30 01:45:06.132989 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:06.132857    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc_odh-ai-gateway-infra(a18da34b-a9cb-4c2b-b7a3-b7921ba4425e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc" podUID="a18da34b-a9cb-4c2b-b7a3-b7921ba4425e"
Oct 30 01:45:06.134267 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:06.134246    4864 scope.go:122] "RemoveContainer" containerID="2ded57d0c40b0305f8caa0f2ac57bb5ff71a601d01eb9fd78fe0ed2672ec2ca3"
Oct 30 01:45:06.134590 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:06.134564    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7_odh-ai-gateway-infra(e9245410-0ba2-4d32-abd2-cfa9e692fd78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7" podUID="e9245410-0ba2-4d32-abd2-cfa9e692fd78"
Oct 30 01:45:06.135805 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:06.135782    4864 scope.go:122] "RemoveContainer" containerID="2594f78b2362490bd5c09b7a633add4898af8b5db13fe3150ef7b77791b14df4"
Oct 30 01:45:06.136176 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:06.136148    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll_odh-ai-gateway-infra(241af13e-1e3a-466a-88b4-111de262ca2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll" podUID="241af13e-1e3a-466a-88b4-111de262ca2a"
Oct 30 01:45:06.137449 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:06.137428    4864 scope.go:122] "RemoveContainer" containerID="81c7d2fdf5cedf38a8604be56a3e5ef620374c621107fd17ef7d9a7b111805b6"
Oct 30 01:45:06.137703 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:06.137677    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz_odh-ai-gateway-infra(ba90b336-84e0-41ce-a673-29892a67100e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz" podUID="ba90b336-84e0-41ce-a673-29892a67100e"
Oct 30 01:45:06.139052 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:06.139035    4864 scope.go:122] "RemoveContainer" containerID="21ba6529d04c063fc43e82807868fdaef1ba4d9f127cab67fef125a03ce39e82"
Oct 30 01:45:06.139291 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:06.139269    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp_odh-ai-gateway-infra(946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp" podUID="946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e"
Oct 30 01:45:06.140828 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:06.140811    4864 scope.go:122] "RemoveContainer" containerID="07adc3b3bcaa902c423e5c9a07d2f93efe884ffc6c3f5d95d94ef0eb04cd8fe8"
Oct 30 01:45:06.141112 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:06.141055    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq_odh-ai-gateway-infra(ce4812b0-4108-4d51-98cf-607aab4eebc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq" podUID="ce4812b0-4108-4d51-98cf-607aab4eebc6"
Oct 30 01:45:06.627640 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:06.627599    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd" podUID="92a9c763-b5f0-4932-aa83-83b5afef6de9" containerName="payload-processing" containerID="cri-o://2d488f058c9fe4c567c3669dd98f65849fe8f2caf1a5053fc5e762e9f8e0ce31" gracePeriod=25
Oct 30 01:45:06.653681 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:06.653651    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g" podUID="e167ecf0-dd04-4531-9fbb-d985e9fe5132" containerName="payload-pre-processing" containerID="cri-o://d467f7e0ea7569fc472cb99e53fb3ab6e2435cea4eb907d244f32c13c100483f" gracePeriod=25
Oct 30 01:45:06.946108 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:06.944546    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd"
Oct 30 01:45:07.041034 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:07.040998    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92a9c763-b5f0-4932-aa83-83b5afef6de9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a9c763-b5f0-4932-aa83-83b5afef6de9-plugins-config-volume\") pod \"92a9c763-b5f0-4932-aa83-83b5afef6de9\" (UID: \"92a9c763-b5f0-4932-aa83-83b5afef6de9\") "
Oct 30 01:45:07.041252 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:07.041129    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a9c763-b5f0-4932-aa83-83b5afef6de9-kube-api-access-8tnw7\" (UniqueName: \"kubernetes.io/projected/92a9c763-b5f0-4932-aa83-83b5afef6de9-kube-api-access-8tnw7\") pod \"92a9c763-b5f0-4932-aa83-83b5afef6de9\" (UID: \"92a9c763-b5f0-4932-aa83-83b5afef6de9\") "
Oct 30 01:45:07.041440 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:07.041412    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a9c763-b5f0-4932-aa83-83b5afef6de9-plugins-config-volume" pod "92a9c763-b5f0-4932-aa83-83b5afef6de9" (UID: "92a9c763-b5f0-4932-aa83-83b5afef6de9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:07.041750 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:07.041734    4864 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a9c763-b5f0-4932-aa83-83b5afef6de9-plugins-config-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:07.043940 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:07.043906    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a9c763-b5f0-4932-aa83-83b5afef6de9-kube-api-access-8tnw7" pod "92a9c763-b5f0-4932-aa83-83b5afef6de9" (UID: "92a9c763-b5f0-4932-aa83-83b5afef6de9"). InnerVolumeSpecName "kube-api-access-8tnw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:07.143621 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:07.143536    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tnw7\" (UniqueName: \"kubernetes.io/projected/92a9c763-b5f0-4932-aa83-83b5afef6de9-kube-api-access-8tnw7\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:07.149619 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:07.149586    4864 generic.go:356] "Generic (PLEG): container finished" podID="92a9c763-b5f0-4932-aa83-83b5afef6de9" containerID="2d488f058c9fe4c567c3669dd98f65849fe8f2caf1a5053fc5e762e9f8e0ce31" exitCode=0
Oct 30 01:45:07.149766 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:07.149638    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd"
Oct 30 01:45:07.149766 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:07.149675    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd" event={"ID":"92a9c763-b5f0-4932-aa83-83b5afef6de9","Type":"ContainerDied","Data":"2d488f058c9fe4c567c3669dd98f65849fe8f2caf1a5053fc5e762e9f8e0ce31"}
Oct 30 01:45:07.149766 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:07.149737    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd" event={"ID":"92a9c763-b5f0-4932-aa83-83b5afef6de9","Type":"ContainerDied","Data":"7b8d374fd60bb51ce7dd4b173329338a2c05566376d525b2109715b7ca108759"}
Oct 30 01:45:07.149766 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:07.149762    4864 scope.go:122] "RemoveContainer" containerID="2d488f058c9fe4c567c3669dd98f65849fe8f2caf1a5053fc5e762e9f8e0ce31"
Oct 30 01:45:07.160459 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:07.160442    4864 scope.go:122] "RemoveContainer" containerID="2d488f058c9fe4c567c3669dd98f65849fe8f2caf1a5053fc5e762e9f8e0ce31"
Oct 30 01:45:07.160730 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:07.160712    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d488f058c9fe4c567c3669dd98f65849fe8f2caf1a5053fc5e762e9f8e0ce31\": container with ID starting with 2d488f058c9fe4c567c3669dd98f65849fe8f2caf1a5053fc5e762e9f8e0ce31 not found: ID does not exist" containerID="2d488f058c9fe4c567c3669dd98f65849fe8f2caf1a5053fc5e762e9f8e0ce31"
Oct 30 01:45:07.160778 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:07.160738    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d488f058c9fe4c567c3669dd98f65849fe8f2caf1a5053fc5e762e9f8e0ce31"} err="failed to get container status \"2d488f058c9fe4c567c3669dd98f65849fe8f2caf1a5053fc5e762e9f8e0ce31\": rpc error: code = NotFound desc = could not find container \"2d488f058c9fe4c567c3669dd98f65849fe8f2caf1a5053fc5e762e9f8e0ce31\": container with ID starting with 2d488f058c9fe4c567c3669dd98f65849fe8f2caf1a5053fc5e762e9f8e0ce31 not found: ID does not exist"
Oct 30 01:45:07.216949 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:07.216921    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd"]
Oct 30 01:45:07.230173 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:07.230146    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-73423b-96c6d6988-5jgwd"]
Oct 30 01:45:07.795800 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:07.795769    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92a9c763-b5f0-4932-aa83-83b5afef6de9" path="/var/lib/kubelet/pods/92a9c763-b5f0-4932-aa83-83b5afef6de9/volumes"
Oct 30 01:45:14.786792 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:14.786710    4864 scope.go:122] "RemoveContainer" containerID="86780da58dca1f3094e735362b441ec88983aadf62146eb169a4913c2e7d5d76"
Oct 30 01:45:14.787258 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:14.786798    4864 scope.go:122] "RemoveContainer" containerID="48a87dc257cbc36b0587dbbe59b2f74c9017e369b1b0c1e80f5398d93df71543"
Oct 30 01:45:15.209801 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:15.209761    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc" event={"ID":"a18da34b-a9cb-4c2b-b7a3-b7921ba4425e","Type":"ContainerStarted","Data":"812044d0eeb83b2989778d59dc03ba4ecc10cdcefcb0a73086f572dfa9892a06"}
Oct 30 01:45:15.211804 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:15.211779    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7" event={"ID":"d9774ce4-b064-4823-a4dc-e915ff4820f0","Type":"ContainerStarted","Data":"5af4cd3db7d8cf50694910ce71c574f63d2c45d5618a23828d8a755fd5d83d41"}
Oct 30 01:45:15.786918 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:15.786893    4864 scope.go:122] "RemoveContainer" containerID="2ded57d0c40b0305f8caa0f2ac57bb5ff71a601d01eb9fd78fe0ed2672ec2ca3"
Oct 30 01:45:16.221208 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:16.221111    4864 generic.go:356] "Generic (PLEG): container finished" podID="d9774ce4-b064-4823-a4dc-e915ff4820f0" containerID="5af4cd3db7d8cf50694910ce71c574f63d2c45d5618a23828d8a755fd5d83d41" exitCode=60
Oct 30 01:45:16.221208 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:16.221187    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7" event={"ID":"d9774ce4-b064-4823-a4dc-e915ff4820f0","Type":"ContainerDied","Data":"5af4cd3db7d8cf50694910ce71c574f63d2c45d5618a23828d8a755fd5d83d41"}
Oct 30 01:45:16.221430 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:16.221258    4864 scope.go:122] "RemoveContainer" containerID="48a87dc257cbc36b0587dbbe59b2f74c9017e369b1b0c1e80f5398d93df71543"
Oct 30 01:45:16.221667 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:16.221630    4864 scope.go:122] "RemoveContainer" containerID="5af4cd3db7d8cf50694910ce71c574f63d2c45d5618a23828d8a755fd5d83d41"
Oct 30 01:45:16.222015 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:16.221987    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7_odh-ai-gateway-infra(d9774ce4-b064-4823-a4dc-e915ff4820f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7" podUID="d9774ce4-b064-4823-a4dc-e915ff4820f0"
Oct 30 01:45:16.223612 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:16.223579    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7" event={"ID":"e9245410-0ba2-4d32-abd2-cfa9e692fd78","Type":"ContainerStarted","Data":"a61797aae6d984d374df2732a245b812b7524b32cddfbe97e498582b52b7af4a"}
Oct 30 01:45:16.225643 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:16.225622    4864 generic.go:356] "Generic (PLEG): container finished" podID="a18da34b-a9cb-4c2b-b7a3-b7921ba4425e" containerID="812044d0eeb83b2989778d59dc03ba4ecc10cdcefcb0a73086f572dfa9892a06" exitCode=60
Oct 30 01:45:16.225744 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:16.225658    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc" event={"ID":"a18da34b-a9cb-4c2b-b7a3-b7921ba4425e","Type":"ContainerDied","Data":"812044d0eeb83b2989778d59dc03ba4ecc10cdcefcb0a73086f572dfa9892a06"}
Oct 30 01:45:16.226015 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:16.225998    4864 scope.go:122] "RemoveContainer" containerID="812044d0eeb83b2989778d59dc03ba4ecc10cdcefcb0a73086f572dfa9892a06"
Oct 30 01:45:16.226337 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:16.226317    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc_odh-ai-gateway-infra(a18da34b-a9cb-4c2b-b7a3-b7921ba4425e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc" podUID="a18da34b-a9cb-4c2b-b7a3-b7921ba4425e"
Oct 30 01:45:16.235123 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:16.235102    4864 scope.go:122] "RemoveContainer" containerID="86780da58dca1f3094e735362b441ec88983aadf62146eb169a4913c2e7d5d76"
Oct 30 01:45:16.269386 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:16.269357    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc"]
Oct 30 01:45:16.315427 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:16.315397    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7"]
Oct 30 01:45:16.553845 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:16.553762    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g" podUID="e167ecf0-dd04-4531-9fbb-d985e9fe5132" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:45:16.786544 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:16.786517    4864 scope.go:122] "RemoveContainer" containerID="07adc3b3bcaa902c423e5c9a07d2f93efe884ffc6c3f5d95d94ef0eb04cd8fe8"
Oct 30 01:45:17.238661 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:17.238631    4864 generic.go:356] "Generic (PLEG): container finished" podID="e9245410-0ba2-4d32-abd2-cfa9e692fd78" containerID="a61797aae6d984d374df2732a245b812b7524b32cddfbe97e498582b52b7af4a" exitCode=60
Oct 30 01:45:17.239137 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:17.238701    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7" event={"ID":"e9245410-0ba2-4d32-abd2-cfa9e692fd78","Type":"ContainerDied","Data":"a61797aae6d984d374df2732a245b812b7524b32cddfbe97e498582b52b7af4a"}
Oct 30 01:45:17.239137 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:17.238794    4864 scope.go:122] "RemoveContainer" containerID="2ded57d0c40b0305f8caa0f2ac57bb5ff71a601d01eb9fd78fe0ed2672ec2ca3"
Oct 30 01:45:17.239247 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:17.239221    4864 scope.go:122] "RemoveContainer" containerID="a61797aae6d984d374df2732a245b812b7524b32cddfbe97e498582b52b7af4a"
Oct 30 01:45:17.239652 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:17.239626    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7_odh-ai-gateway-infra(e9245410-0ba2-4d32-abd2-cfa9e692fd78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7" podUID="e9245410-0ba2-4d32-abd2-cfa9e692fd78"
Oct 30 01:45:17.241242 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:17.241220    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq" event={"ID":"ce4812b0-4108-4d51-98cf-607aab4eebc6","Type":"ContainerStarted","Data":"40112004850d1a712329176d426b071a7285aa8fc61c29d30292351e425502ab"}
Oct 30 01:45:17.349707 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:17.349639    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7"]
Oct 30 01:45:17.417760 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:17.417730    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc"
Oct 30 01:45:17.477040 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:17.477007    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7"
Oct 30 01:45:17.506805 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:17.506725    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a18da34b-a9cb-4c2b-b7a3-b7921ba4425e-kube-api-access-m748d\" (UniqueName: \"kubernetes.io/projected/a18da34b-a9cb-4c2b-b7a3-b7921ba4425e-kube-api-access-m748d\") pod \"a18da34b-a9cb-4c2b-b7a3-b7921ba4425e\" (UID: \"a18da34b-a9cb-4c2b-b7a3-b7921ba4425e\") "
Oct 30 01:45:17.506959 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:17.506836    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a18da34b-a9cb-4c2b-b7a3-b7921ba4425e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18da34b-a9cb-4c2b-b7a3-b7921ba4425e-maas-api-service-ca\") pod \"a18da34b-a9cb-4c2b-b7a3-b7921ba4425e\" (UID: \"a18da34b-a9cb-4c2b-b7a3-b7921ba4425e\") "
Oct 30 01:45:17.507026 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:17.506988    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9774ce4-b064-4823-a4dc-e915ff4820f0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9774ce4-b064-4823-a4dc-e915ff4820f0-maas-api-service-ca\") pod \"d9774ce4-b064-4823-a4dc-e915ff4820f0\" (UID: \"d9774ce4-b064-4823-a4dc-e915ff4820f0\") "
Oct 30 01:45:17.507163 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:17.507140    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9774ce4-b064-4823-a4dc-e915ff4820f0-kube-api-access-gfpxt\" (UniqueName: \"kubernetes.io/projected/d9774ce4-b064-4823-a4dc-e915ff4820f0-kube-api-access-gfpxt\") pod \"d9774ce4-b064-4823-a4dc-e915ff4820f0\" (UID: \"d9774ce4-b064-4823-a4dc-e915ff4820f0\") "
Oct 30 01:45:17.507271 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:17.507249    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a18da34b-a9cb-4c2b-b7a3-b7921ba4425e-maas-api-service-ca" pod "a18da34b-a9cb-4c2b-b7a3-b7921ba4425e" (UID: "a18da34b-a9cb-4c2b-b7a3-b7921ba4425e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:17.507511 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:17.507482    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9774ce4-b064-4823-a4dc-e915ff4820f0-maas-api-service-ca" pod "d9774ce4-b064-4823-a4dc-e915ff4820f0" (UID: "d9774ce4-b064-4823-a4dc-e915ff4820f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:17.507831 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:17.507809    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18da34b-a9cb-4c2b-b7a3-b7921ba4425e-maas-api-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:17.507919 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:17.507836    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9774ce4-b064-4823-a4dc-e915ff4820f0-maas-api-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:17.509557 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:17.509534    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18da34b-a9cb-4c2b-b7a3-b7921ba4425e-kube-api-access-m748d" pod "a18da34b-a9cb-4c2b-b7a3-b7921ba4425e" (UID: "a18da34b-a9cb-4c2b-b7a3-b7921ba4425e"). InnerVolumeSpecName "kube-api-access-m748d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:17.509654 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:17.509570    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9774ce4-b064-4823-a4dc-e915ff4820f0-kube-api-access-gfpxt" pod "d9774ce4-b064-4823-a4dc-e915ff4820f0" (UID: "d9774ce4-b064-4823-a4dc-e915ff4820f0"). InnerVolumeSpecName "kube-api-access-gfpxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:17.609058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:17.609019    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfpxt\" (UniqueName: \"kubernetes.io/projected/d9774ce4-b064-4823-a4dc-e915ff4820f0-kube-api-access-gfpxt\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:17.609058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:17.609055    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m748d\" (UniqueName: \"kubernetes.io/projected/a18da34b-a9cb-4c2b-b7a3-b7921ba4425e-kube-api-access-m748d\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:17.791162 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:17.791045    4864 scope.go:122] "RemoveContainer" containerID="21ba6529d04c063fc43e82807868fdaef1ba4d9f127cab67fef125a03ce39e82"
Oct 30 01:45:17.791332 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:17.791214    4864 scope.go:122] "RemoveContainer" containerID="2594f78b2362490bd5c09b7a633add4898af8b5db13fe3150ef7b77791b14df4"
Oct 30 01:45:18.251595 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:18.251563    4864 generic.go:356] "Generic (PLEG): container finished" podID="ce4812b0-4108-4d51-98cf-607aab4eebc6" containerID="40112004850d1a712329176d426b071a7285aa8fc61c29d30292351e425502ab" exitCode=60
Oct 30 01:45:18.252052 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:18.251632    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq" event={"ID":"ce4812b0-4108-4d51-98cf-607aab4eebc6","Type":"ContainerDied","Data":"40112004850d1a712329176d426b071a7285aa8fc61c29d30292351e425502ab"}
Oct 30 01:45:18.252052 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:18.251696    4864 scope.go:122] "RemoveContainer" containerID="07adc3b3bcaa902c423e5c9a07d2f93efe884ffc6c3f5d95d94ef0eb04cd8fe8"
Oct 30 01:45:18.252250 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:18.252155    4864 scope.go:122] "RemoveContainer" containerID="40112004850d1a712329176d426b071a7285aa8fc61c29d30292351e425502ab"
Oct 30 01:45:18.252526 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:18.252498    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq_odh-ai-gateway-infra(ce4812b0-4108-4d51-98cf-607aab4eebc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq" podUID="ce4812b0-4108-4d51-98cf-607aab4eebc6"
Oct 30 01:45:18.253408 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:18.253384    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc"
Oct 30 01:45:18.253508 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:18.253405    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc" event={"ID":"a18da34b-a9cb-4c2b-b7a3-b7921ba4425e","Type":"ContainerDied","Data":"5dd045246988bb04aee36ceffa04fbe91ae2e74be6b03223c49d88893836c84f"}
Oct 30 01:45:18.254729 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:18.254712    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7"
Oct 30 01:45:18.254838 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:18.254741    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7" event={"ID":"d9774ce4-b064-4823-a4dc-e915ff4820f0","Type":"ContainerDied","Data":"e2c9297f27797650d58fab5cb0906a7673c0f93d31883bda8e69017c3b3b4005"}
Oct 30 01:45:18.259398 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:18.259367    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll" event={"ID":"241af13e-1e3a-466a-88b4-111de262ca2a","Type":"ContainerStarted","Data":"e5ee22cc456afeb015541431012fbc4dc1de0a7b38622471e7e79973b03b5eff"}
Oct 30 01:45:18.271456 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:18.271423    4864 scope.go:122] "RemoveContainer" containerID="812044d0eeb83b2989778d59dc03ba4ecc10cdcefcb0a73086f572dfa9892a06"
Oct 30 01:45:18.282732 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:18.282711    4864 scope.go:122] "RemoveContainer" containerID="5af4cd3db7d8cf50694910ce71c574f63d2c45d5618a23828d8a755fd5d83d41"
Oct 30 01:45:18.325012 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:18.324930    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq"]
Oct 30 01:45:18.365478 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:18.365447    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc"]
Oct 30 01:45:18.376916 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:18.376862    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888745-g7zzc"]
Oct 30 01:45:18.408775 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:18.408749    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7"]
Oct 30 01:45:18.413347 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:18.413320    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-72ed13-29888745-mqwn7"]
Oct 30 01:45:18.491638 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:18.491609    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7"
Oct 30 01:45:18.526029 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:18.525995    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9245410-0ba2-4d32-abd2-cfa9e692fd78-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9245410-0ba2-4d32-abd2-cfa9e692fd78-maas-api-service-ca\") pod \"e9245410-0ba2-4d32-abd2-cfa9e692fd78\" (UID: \"e9245410-0ba2-4d32-abd2-cfa9e692fd78\") "
Oct 30 01:45:18.526344 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:18.526323    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9245410-0ba2-4d32-abd2-cfa9e692fd78-kube-api-access-2zcmn\" (UniqueName: \"kubernetes.io/projected/e9245410-0ba2-4d32-abd2-cfa9e692fd78-kube-api-access-2zcmn\") pod \"e9245410-0ba2-4d32-abd2-cfa9e692fd78\" (UID: \"e9245410-0ba2-4d32-abd2-cfa9e692fd78\") "
Oct 30 01:45:18.526443 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:18.526371    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9245410-0ba2-4d32-abd2-cfa9e692fd78-maas-api-service-ca" pod "e9245410-0ba2-4d32-abd2-cfa9e692fd78" (UID: "e9245410-0ba2-4d32-abd2-cfa9e692fd78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:18.526751 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:18.526732    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9245410-0ba2-4d32-abd2-cfa9e692fd78-maas-api-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:18.528506 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:18.528478    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9245410-0ba2-4d32-abd2-cfa9e692fd78-kube-api-access-2zcmn" pod "e9245410-0ba2-4d32-abd2-cfa9e692fd78" (UID: "e9245410-0ba2-4d32-abd2-cfa9e692fd78"). InnerVolumeSpecName "kube-api-access-2zcmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:18.627655 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:18.627576    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zcmn\" (UniqueName: \"kubernetes.io/projected/e9245410-0ba2-4d32-abd2-cfa9e692fd78-kube-api-access-2zcmn\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:18.786180 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:18.786148    4864 scope.go:122] "RemoveContainer" containerID="81c7d2fdf5cedf38a8604be56a3e5ef620374c621107fd17ef7d9a7b111805b6"
Oct 30 01:45:19.043659 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:19.043621    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:45:19.274029 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.273930    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7" event={"ID":"e9245410-0ba2-4d32-abd2-cfa9e692fd78","Type":"ContainerDied","Data":"312c995b681193ee30e36214b9d408bcfc1b0ea262a7eeafe4b983a7be897f58"}
Oct 30 01:45:19.274588 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.274047    4864 scope.go:122] "RemoveContainer" containerID="a61797aae6d984d374df2732a245b812b7524b32cddfbe97e498582b52b7af4a"
Oct 30 01:45:19.274682 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.273955    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7"
Oct 30 01:45:19.277713 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.277682    4864 generic.go:356] "Generic (PLEG): container finished" podID="241af13e-1e3a-466a-88b4-111de262ca2a" containerID="e5ee22cc456afeb015541431012fbc4dc1de0a7b38622471e7e79973b03b5eff" exitCode=60
Oct 30 01:45:19.277840 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.277821    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll" event={"ID":"241af13e-1e3a-466a-88b4-111de262ca2a","Type":"ContainerDied","Data":"e5ee22cc456afeb015541431012fbc4dc1de0a7b38622471e7e79973b03b5eff"}
Oct 30 01:45:19.278351 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.278330    4864 scope.go:122] "RemoveContainer" containerID="e5ee22cc456afeb015541431012fbc4dc1de0a7b38622471e7e79973b03b5eff"
Oct 30 01:45:19.282599 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:19.278701    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll_odh-ai-gateway-infra(241af13e-1e3a-466a-88b4-111de262ca2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll" podUID="241af13e-1e3a-466a-88b4-111de262ca2a"
Oct 30 01:45:19.282599 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.281241    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz" event={"ID":"ba90b336-84e0-41ce-a673-29892a67100e","Type":"ContainerStarted","Data":"8b4e1658cb855173d4e495db49a6ef606cabfe4ae0638d351f704794859d7041"}
Oct 30 01:45:19.284029 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.284002    4864 generic.go:356] "Generic (PLEG): container finished" podID="946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e" containerID="b1a543d0d3e38b173a060a618f8b6857f1556d71e26036e95f6a97793ecafdee" exitCode=60
Oct 30 01:45:19.284149 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.284134    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp" event={"ID":"946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e","Type":"ContainerDied","Data":"b1a543d0d3e38b173a060a618f8b6857f1556d71e26036e95f6a97793ecafdee"}
Oct 30 01:45:19.284437 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.284422    4864 scope.go:122] "RemoveContainer" containerID="b1a543d0d3e38b173a060a618f8b6857f1556d71e26036e95f6a97793ecafdee"
Oct 30 01:45:19.284670 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:19.284647    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp_odh-ai-gateway-infra(946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp" podUID="946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e"
Oct 30 01:45:19.295318 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.293751    4864 scope.go:122] "RemoveContainer" containerID="2594f78b2362490bd5c09b7a633add4898af8b5db13fe3150ef7b77791b14df4"
Oct 30 01:45:19.310634 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.310610    4864 scope.go:122] "RemoveContainer" containerID="21ba6529d04c063fc43e82807868fdaef1ba4d9f127cab67fef125a03ce39e82"
Oct 30 01:45:19.350708 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.350678    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll"]
Oct 30 01:45:19.489827 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.489798    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq"
Oct 30 01:45:19.498330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.498305    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7"]
Oct 30 01:45:19.508616 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.508589    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888745-h6qq7"]
Oct 30 01:45:19.540929 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.540852    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce4812b0-4108-4d51-98cf-607aab4eebc6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce4812b0-4108-4d51-98cf-607aab4eebc6-maas-api-service-ca\") pod \"ce4812b0-4108-4d51-98cf-607aab4eebc6\" (UID: \"ce4812b0-4108-4d51-98cf-607aab4eebc6\") "
Oct 30 01:45:19.540929 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.540885    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce4812b0-4108-4d51-98cf-607aab4eebc6-kube-api-access-rwm5p\" (UniqueName: \"kubernetes.io/projected/ce4812b0-4108-4d51-98cf-607aab4eebc6-kube-api-access-rwm5p\") pod \"ce4812b0-4108-4d51-98cf-607aab4eebc6\" (UID: \"ce4812b0-4108-4d51-98cf-607aab4eebc6\") "
Oct 30 01:45:19.541299 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.541276    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce4812b0-4108-4d51-98cf-607aab4eebc6-maas-api-service-ca" pod "ce4812b0-4108-4d51-98cf-607aab4eebc6" (UID: "ce4812b0-4108-4d51-98cf-607aab4eebc6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:19.541517 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.541501    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce4812b0-4108-4d51-98cf-607aab4eebc6-maas-api-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:19.543299 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.543272    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce4812b0-4108-4d51-98cf-607aab4eebc6-kube-api-access-rwm5p" pod "ce4812b0-4108-4d51-98cf-607aab4eebc6" (UID: "ce4812b0-4108-4d51-98cf-607aab4eebc6"). InnerVolumeSpecName "kube-api-access-rwm5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:19.643210 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.643174    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwm5p\" (UniqueName: \"kubernetes.io/projected/ce4812b0-4108-4d51-98cf-607aab4eebc6-kube-api-access-rwm5p\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:19.795313 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.795234    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a18da34b-a9cb-4c2b-b7a3-b7921ba4425e" path="/var/lib/kubelet/pods/a18da34b-a9cb-4c2b-b7a3-b7921ba4425e/volumes"
Oct 30 01:45:19.795738 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.795724    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9774ce4-b064-4823-a4dc-e915ff4820f0" path="/var/lib/kubelet/pods/d9774ce4-b064-4823-a4dc-e915ff4820f0/volumes"
Oct 30 01:45:19.796161 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:19.796145    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9245410-0ba2-4d32-abd2-cfa9e692fd78" path="/var/lib/kubelet/pods/e9245410-0ba2-4d32-abd2-cfa9e692fd78/volumes"
Oct 30 01:45:20.300013 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.299984    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq"
Oct 30 01:45:20.300514 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.299985    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq" event={"ID":"ce4812b0-4108-4d51-98cf-607aab4eebc6","Type":"ContainerDied","Data":"232b6bee313f406f1e21f6d714f8713bbdbc7e800a54a990b5fd92e06fe633ee"}
Oct 30 01:45:20.300514 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.300139    4864 scope.go:122] "RemoveContainer" containerID="40112004850d1a712329176d426b071a7285aa8fc61c29d30292351e425502ab"
Oct 30 01:45:20.305959 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.305930    4864 generic.go:356] "Generic (PLEG): container finished" podID="ba90b336-84e0-41ce-a673-29892a67100e" containerID="8b4e1658cb855173d4e495db49a6ef606cabfe4ae0638d351f704794859d7041" exitCode=60
Oct 30 01:45:20.306047 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.305998    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz" event={"ID":"ba90b336-84e0-41ce-a673-29892a67100e","Type":"ContainerDied","Data":"8b4e1658cb855173d4e495db49a6ef606cabfe4ae0638d351f704794859d7041"}
Oct 30 01:45:20.306537 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.306518    4864 scope.go:122] "RemoveContainer" containerID="8b4e1658cb855173d4e495db49a6ef606cabfe4ae0638d351f704794859d7041"
Oct 30 01:45:20.306905 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:20.306883    4864 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz_odh-ai-gateway-infra(ba90b336-84e0-41ce-a673-29892a67100e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz" podUID="ba90b336-84e0-41ce-a673-29892a67100e"
Oct 30 01:45:20.312878 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.312850    4864 scope.go:122] "RemoveContainer" containerID="81c7d2fdf5cedf38a8604be56a3e5ef620374c621107fd17ef7d9a7b111805b6"
Oct 30 01:45:20.354165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.354136    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq"]
Oct 30 01:45:20.357972 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.357938    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888745-jvxjq"]
Oct 30 01:45:20.393459 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.393421    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp"]
Oct 30 01:45:20.476242 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.475964    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz"]
Oct 30 01:45:20.488128 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.488102    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll"
Oct 30 01:45:20.560687 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.560600    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/241af13e-1e3a-466a-88b4-111de262ca2a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/241af13e-1e3a-466a-88b4-111de262ca2a-maas-api-service-ca\") pod \"241af13e-1e3a-466a-88b4-111de262ca2a\" (UID: \"241af13e-1e3a-466a-88b4-111de262ca2a\") "
Oct 30 01:45:20.560835 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.560724    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/241af13e-1e3a-466a-88b4-111de262ca2a-kube-api-access-scxp4\" (UniqueName: \"kubernetes.io/projected/241af13e-1e3a-466a-88b4-111de262ca2a-kube-api-access-scxp4\") pod \"241af13e-1e3a-466a-88b4-111de262ca2a\" (UID: \"241af13e-1e3a-466a-88b4-111de262ca2a\") "
Oct 30 01:45:20.561019 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.560993    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/241af13e-1e3a-466a-88b4-111de262ca2a-maas-api-service-ca" pod "241af13e-1e3a-466a-88b4-111de262ca2a" (UID: "241af13e-1e3a-466a-88b4-111de262ca2a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:20.561404 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.561384    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/241af13e-1e3a-466a-88b4-111de262ca2a-maas-api-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:20.563624 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.563599    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241af13e-1e3a-466a-88b4-111de262ca2a-kube-api-access-scxp4" pod "241af13e-1e3a-466a-88b4-111de262ca2a" (UID: "241af13e-1e3a-466a-88b4-111de262ca2a"). InnerVolumeSpecName "kube-api-access-scxp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:20.621281 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.621256    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp"
Oct 30 01:45:20.662181 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.662142    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e-kube-api-access-sbxks\" (UniqueName: \"kubernetes.io/projected/946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e-kube-api-access-sbxks\") pod \"946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e\" (UID: \"946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e\") "
Oct 30 01:45:20.662383 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.662364    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e-maas-api-service-ca\") pod \"946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e\" (UID: \"946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e\") "
Oct 30 01:45:20.662811 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.662780    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e-maas-api-service-ca" pod "946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e" (UID: "946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:20.662811 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.662979    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e-maas-api-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:20.662811 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.662997    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scxp4\" (UniqueName: \"kubernetes.io/projected/241af13e-1e3a-466a-88b4-111de262ca2a-kube-api-access-scxp4\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:20.664642 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.664621    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e-kube-api-access-sbxks" pod "946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e" (UID: "946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e"). InnerVolumeSpecName "kube-api-access-sbxks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:20.764722 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:20.764681    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbxks\" (UniqueName: \"kubernetes.io/projected/946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e-kube-api-access-sbxks\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:21.316883 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:21.316853    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll"
Oct 30 01:45:21.317396 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:21.316885    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll" event={"ID":"241af13e-1e3a-466a-88b4-111de262ca2a","Type":"ContainerDied","Data":"b6115e06d6d0b866464776b720f484c87c92c6ba2a25280cbc28b475ba5ea90e"}
Oct 30 01:45:21.317396 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:21.316948    4864 scope.go:122] "RemoveContainer" containerID="e5ee22cc456afeb015541431012fbc4dc1de0a7b38622471e7e79973b03b5eff"
Oct 30 01:45:21.320593 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:21.320569    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp" event={"ID":"946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e","Type":"ContainerDied","Data":"126128261ac332f9e325a1b4334f7beb468b134af9b55c5147bd972ab95f9ec9"}
Oct 30 01:45:21.320593 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:21.320584    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp"
Oct 30 01:45:21.331027 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:21.331006    4864 scope.go:122] "RemoveContainer" containerID="b1a543d0d3e38b173a060a618f8b6857f1556d71e26036e95f6a97793ecafdee"
Oct 30 01:45:21.408252 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:21.407600    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp"]
Oct 30 01:45:21.415790 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:21.415757    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888745-fzxhp"]
Oct 30 01:45:21.454312 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:21.454280    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll"]
Oct 30 01:45:21.458547 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:21.458521    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ae1820-29888745-df7ll"]
Oct 30 01:45:21.492168 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:21.492125    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz"
Oct 30 01:45:21.575887 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:21.575800    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba90b336-84e0-41ce-a673-29892a67100e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba90b336-84e0-41ce-a673-29892a67100e-maas-api-service-ca\") pod \"ba90b336-84e0-41ce-a673-29892a67100e\" (UID: \"ba90b336-84e0-41ce-a673-29892a67100e\") "
Oct 30 01:45:21.576048 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:21.575905    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba90b336-84e0-41ce-a673-29892a67100e-kube-api-access-c6pdx\" (UniqueName: \"kubernetes.io/projected/ba90b336-84e0-41ce-a673-29892a67100e-kube-api-access-c6pdx\") pod \"ba90b336-84e0-41ce-a673-29892a67100e\" (UID: \"ba90b336-84e0-41ce-a673-29892a67100e\") "
Oct 30 01:45:21.576333 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:21.576273    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba90b336-84e0-41ce-a673-29892a67100e-maas-api-service-ca" pod "ba90b336-84e0-41ce-a673-29892a67100e" (UID: "ba90b336-84e0-41ce-a673-29892a67100e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:21.576498 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:21.576476    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba90b336-84e0-41ce-a673-29892a67100e-maas-api-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:21.578248 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:21.578220    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba90b336-84e0-41ce-a673-29892a67100e-kube-api-access-c6pdx" pod "ba90b336-84e0-41ce-a673-29892a67100e" (UID: "ba90b336-84e0-41ce-a673-29892a67100e"). InnerVolumeSpecName "kube-api-access-c6pdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:21.678273 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:21.678240    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6pdx\" (UniqueName: \"kubernetes.io/projected/ba90b336-84e0-41ce-a673-29892a67100e-kube-api-access-c6pdx\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:21.795599 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:21.795562    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="241af13e-1e3a-466a-88b4-111de262ca2a" path="/var/lib/kubelet/pods/241af13e-1e3a-466a-88b4-111de262ca2a/volumes"
Oct 30 01:45:21.796032 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:21.796018    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e" path="/var/lib/kubelet/pods/946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e/volumes"
Oct 30 01:45:21.796435 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:21.796421    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce4812b0-4108-4d51-98cf-607aab4eebc6" path="/var/lib/kubelet/pods/ce4812b0-4108-4d51-98cf-607aab4eebc6/volumes"
Oct 30 01:45:22.328971 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:22.328940    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz"
Oct 30 01:45:22.328971 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:22.328964    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz" event={"ID":"ba90b336-84e0-41ce-a673-29892a67100e","Type":"ContainerDied","Data":"3f983522c904e08d590dbc5660b039e2f210a2f13a00f8cbf5403a830d3ba031"}
Oct 30 01:45:22.329523 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:22.329037    4864 scope.go:122] "RemoveContainer" containerID="8b4e1658cb855173d4e495db49a6ef606cabfe4ae0638d351f704794859d7041"
Oct 30 01:45:22.380431 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:22.380396    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz"]
Oct 30 01:45:22.391199 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:22.390152    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888745-zbmhz"]
Oct 30 01:45:23.795223 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:23.795190    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba90b336-84e0-41ce-a673-29892a67100e" path="/var/lib/kubelet/pods/ba90b336-84e0-41ce-a673-29892a67100e/volumes"
Oct 30 01:45:26.553102 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:26.553036    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g" podUID="e167ecf0-dd04-4531-9fbb-d985e9fe5132" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:45:26.919150 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:26.919125    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g"
Oct 30 01:45:27.064441 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:27.064410    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e167ecf0-dd04-4531-9fbb-d985e9fe5132-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e167ecf0-dd04-4531-9fbb-d985e9fe5132-plugins-config-volume\") pod \"e167ecf0-dd04-4531-9fbb-d985e9fe5132\" (UID: \"e167ecf0-dd04-4531-9fbb-d985e9fe5132\") "
Oct 30 01:45:27.064441 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:27.064447    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e167ecf0-dd04-4531-9fbb-d985e9fe5132-kube-api-access-7hmfr\" (UniqueName: \"kubernetes.io/projected/e167ecf0-dd04-4531-9fbb-d985e9fe5132-kube-api-access-7hmfr\") pod \"e167ecf0-dd04-4531-9fbb-d985e9fe5132\" (UID: \"e167ecf0-dd04-4531-9fbb-d985e9fe5132\") "
Oct 30 01:45:27.064797 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:27.064771    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e167ecf0-dd04-4531-9fbb-d985e9fe5132-plugins-config-volume" pod "e167ecf0-dd04-4531-9fbb-d985e9fe5132" (UID: "e167ecf0-dd04-4531-9fbb-d985e9fe5132"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:27.065092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:27.065064    4864 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e167ecf0-dd04-4531-9fbb-d985e9fe5132-plugins-config-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:27.066656 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:27.066634    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e167ecf0-dd04-4531-9fbb-d985e9fe5132-kube-api-access-7hmfr" pod "e167ecf0-dd04-4531-9fbb-d985e9fe5132" (UID: "e167ecf0-dd04-4531-9fbb-d985e9fe5132"). InnerVolumeSpecName "kube-api-access-7hmfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:27.167097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:27.167040    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hmfr\" (UniqueName: \"kubernetes.io/projected/e167ecf0-dd04-4531-9fbb-d985e9fe5132-kube-api-access-7hmfr\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:27.367671 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:27.367585    4864 generic.go:356] "Generic (PLEG): container finished" podID="e167ecf0-dd04-4531-9fbb-d985e9fe5132" containerID="d467f7e0ea7569fc472cb99e53fb3ab6e2435cea4eb907d244f32c13c100483f" exitCode=0
Oct 30 01:45:27.367671 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:27.367643    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g"
Oct 30 01:45:27.367850 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:27.367681    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g" event={"ID":"e167ecf0-dd04-4531-9fbb-d985e9fe5132","Type":"ContainerDied","Data":"d467f7e0ea7569fc472cb99e53fb3ab6e2435cea4eb907d244f32c13c100483f"}
Oct 30 01:45:27.367850 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:27.367753    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g" event={"ID":"e167ecf0-dd04-4531-9fbb-d985e9fe5132","Type":"ContainerDied","Data":"4f5fe1505b80ae34ab79d448209f49d9fec2b8e626e2ccc19ae8d594017f89f3"}
Oct 30 01:45:27.367850 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:27.367777    4864 scope.go:122] "RemoveContainer" containerID="d467f7e0ea7569fc472cb99e53fb3ab6e2435cea4eb907d244f32c13c100483f"
Oct 30 01:45:27.378801 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:27.378785    4864 scope.go:122] "RemoveContainer" containerID="d467f7e0ea7569fc472cb99e53fb3ab6e2435cea4eb907d244f32c13c100483f"
Oct 30 01:45:27.379049 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:27.379028    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d467f7e0ea7569fc472cb99e53fb3ab6e2435cea4eb907d244f32c13c100483f\": container with ID starting with d467f7e0ea7569fc472cb99e53fb3ab6e2435cea4eb907d244f32c13c100483f not found: ID does not exist" containerID="d467f7e0ea7569fc472cb99e53fb3ab6e2435cea4eb907d244f32c13c100483f"
Oct 30 01:45:27.379157 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:27.379062    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d467f7e0ea7569fc472cb99e53fb3ab6e2435cea4eb907d244f32c13c100483f"} err="failed to get container status \"d467f7e0ea7569fc472cb99e53fb3ab6e2435cea4eb907d244f32c13c100483f\": rpc error: code = NotFound desc = could not find container \"d467f7e0ea7569fc472cb99e53fb3ab6e2435cea4eb907d244f32c13c100483f\": container with ID starting with d467f7e0ea7569fc472cb99e53fb3ab6e2435cea4eb907d244f32c13c100483f not found: ID does not exist"
Oct 30 01:45:27.417121 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:27.417085    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g"]
Oct 30 01:45:27.426878 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:27.426850    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-73423b-7488446755-tz55g"]
Oct 30 01:45:27.795541 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:27.795503    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e167ecf0-dd04-4531-9fbb-d985e9fe5132" path="/var/lib/kubelet/pods/e167ecf0-dd04-4531-9fbb-d985e9fe5132/volumes"
Oct 30 01:45:29.137502 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:29.137470    4864 scope.go:122] "RemoveContainer" containerID="b98fb73783afe0ef43e59d1762ac0e2d688ff51d5f89c1b00c467d8936451faf"
Oct 30 01:45:49.111103 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:49.099136    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:45:53.678583 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:53.678549    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"]
Oct 30 01:45:53.679418 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:53.679375    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf" podUID="b6897e74-0146-4ff4-89e3-4f1655fd8f35" containerName="istio-proxy" containerID="cri-o://94b1b4d9aa274b8148435ec9dab9997eb6b5bad0140349ab75555a3ca87be763" gracePeriod=30
Oct 30 01:45:57.232441 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:57.232405    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"]
Oct 30 01:45:57.232898 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:57.232682    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85" podUID="31f94d41-d691-4cdc-a24b-fc976bc9e1c8" containerName="istio-proxy" containerID="cri-o://86e6aa31c29cb1847eb0cff35f1e78063a74764aaa4b2c347a3bb959cc1d5498" gracePeriod=30
Oct 30 01:45:59.002816 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.002782    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:45:59.046959 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.046929    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6897e74-0146-4ff4-89e3-4f1655fd8f35-kube-api-access-rg7jz\" (UniqueName: \"kubernetes.io/projected/b6897e74-0146-4ff4-89e3-4f1655fd8f35-kube-api-access-rg7jz\") pod \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") "
Oct 30 01:45:59.047140 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.046975    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-podinfo\") pod \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") "
Oct 30 01:45:59.047140 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.047017    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-workload-certs\") pod \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") "
Oct 30 01:45:59.047140 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.047054    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-token\" (UniqueName: \"kubernetes.io/projected/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-token\") pod \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") "
Oct 30 01:45:59.047320 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.047139    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-envoy\") pod \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") "
Oct 30 01:45:59.047320 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.047256    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-credential-socket\") pod \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") "
Oct 30 01:45:59.047320 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.047284    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-data\") pod \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") "
Oct 30 01:45:59.047486 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.047364    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-workload-socket\") pod \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") "
Oct 30 01:45:59.047486 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.047361    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-workload-certs" pod "b6897e74-0146-4ff4-89e3-4f1655fd8f35" (UID: "b6897e74-0146-4ff4-89e3-4f1655fd8f35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:45:59.047486 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.047414    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istiod-ca-cert\") pod \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\" (UID: \"b6897e74-0146-4ff4-89e3-4f1655fd8f35\") "
Oct 30 01:45:59.047659 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.047602    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-data" pod "b6897e74-0146-4ff4-89e3-4f1655fd8f35" (UID: "b6897e74-0146-4ff4-89e3-4f1655fd8f35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:45:59.047659 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.047623    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-credential-socket" pod "b6897e74-0146-4ff4-89e3-4f1655fd8f35" (UID: "b6897e74-0146-4ff4-89e3-4f1655fd8f35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:45:59.047848 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.047825    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-workload-socket" pod "b6897e74-0146-4ff4-89e3-4f1655fd8f35" (UID: "b6897e74-0146-4ff4-89e3-4f1655fd8f35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:45:59.048116 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.048064    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istiod-ca-cert" pod "b6897e74-0146-4ff4-89e3-4f1655fd8f35" (UID: "b6897e74-0146-4ff4-89e3-4f1655fd8f35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:59.048302 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.048279    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:59.048376 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.048308    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:59.048376 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.048323    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:59.048376 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.048336    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:59.048376 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.048350    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:59.049362 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.049340    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-podinfo" pod "b6897e74-0146-4ff4-89e3-4f1655fd8f35" (UID: "b6897e74-0146-4ff4-89e3-4f1655fd8f35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:45:59.049470 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.049429    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-envoy" pod "b6897e74-0146-4ff4-89e3-4f1655fd8f35" (UID: "b6897e74-0146-4ff4-89e3-4f1655fd8f35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:45:59.049549 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.049528    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6897e74-0146-4ff4-89e3-4f1655fd8f35-kube-api-access-rg7jz" pod "b6897e74-0146-4ff4-89e3-4f1655fd8f35" (UID: "b6897e74-0146-4ff4-89e3-4f1655fd8f35"). InnerVolumeSpecName "kube-api-access-rg7jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:59.049608 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.049556    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-token" pod "b6897e74-0146-4ff4-89e3-4f1655fd8f35" (UID: "b6897e74-0146-4ff4-89e3-4f1655fd8f35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:59.159417 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.159370    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg7jz\" (UniqueName: \"kubernetes.io/projected/b6897e74-0146-4ff4-89e3-4f1655fd8f35-kube-api-access-rg7jz\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:59.159417 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.159405    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:59.159706 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.159437    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:59.159706 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.159451    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6897e74-0146-4ff4-89e3-4f1655fd8f35-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:45:59.714984 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.714934    4864 generic.go:356] "Generic (PLEG): container finished" podID="b6897e74-0146-4ff4-89e3-4f1655fd8f35" containerID="94b1b4d9aa274b8148435ec9dab9997eb6b5bad0140349ab75555a3ca87be763" exitCode=0
Oct 30 01:45:59.715200 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.714997    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf" event={"ID":"b6897e74-0146-4ff4-89e3-4f1655fd8f35","Type":"ContainerDied","Data":"94b1b4d9aa274b8148435ec9dab9997eb6b5bad0140349ab75555a3ca87be763"}
Oct 30 01:45:59.715200 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.715038    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"
Oct 30 01:45:59.715200 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.715091    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf" event={"ID":"b6897e74-0146-4ff4-89e3-4f1655fd8f35","Type":"ContainerDied","Data":"3f6904a58421c3e640a211f5130f7c0345b318800876b44814bf01bf7cf80e19"}
Oct 30 01:45:59.715200 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.715126    4864 scope.go:122] "RemoveContainer" containerID="94b1b4d9aa274b8148435ec9dab9997eb6b5bad0140349ab75555a3ca87be763"
Oct 30 01:45:59.727355 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.727337    4864 scope.go:122] "RemoveContainer" containerID="cecb3050ac1f23091197f927ec5c3a22172c92b96bd97ebb0baf58fcdc12ae4a"
Oct 30 01:45:59.737422 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.737405    4864 scope.go:122] "RemoveContainer" containerID="94b1b4d9aa274b8148435ec9dab9997eb6b5bad0140349ab75555a3ca87be763"
Oct 30 01:45:59.737707 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:59.737688    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b1b4d9aa274b8148435ec9dab9997eb6b5bad0140349ab75555a3ca87be763\": container with ID starting with 94b1b4d9aa274b8148435ec9dab9997eb6b5bad0140349ab75555a3ca87be763 not found: ID does not exist" containerID="94b1b4d9aa274b8148435ec9dab9997eb6b5bad0140349ab75555a3ca87be763"
Oct 30 01:45:59.737825 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.737719    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b1b4d9aa274b8148435ec9dab9997eb6b5bad0140349ab75555a3ca87be763"} err="failed to get container status \"94b1b4d9aa274b8148435ec9dab9997eb6b5bad0140349ab75555a3ca87be763\": rpc error: code = NotFound desc = could not find container \"94b1b4d9aa274b8148435ec9dab9997eb6b5bad0140349ab75555a3ca87be763\": container with ID starting with 94b1b4d9aa274b8148435ec9dab9997eb6b5bad0140349ab75555a3ca87be763 not found: ID does not exist"
Oct 30 01:45:59.737825 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.737746    4864 scope.go:122] "RemoveContainer" containerID="cecb3050ac1f23091197f927ec5c3a22172c92b96bd97ebb0baf58fcdc12ae4a"
Oct 30 01:45:59.737984 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:45:59.737966    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cecb3050ac1f23091197f927ec5c3a22172c92b96bd97ebb0baf58fcdc12ae4a\": container with ID starting with cecb3050ac1f23091197f927ec5c3a22172c92b96bd97ebb0baf58fcdc12ae4a not found: ID does not exist" containerID="cecb3050ac1f23091197f927ec5c3a22172c92b96bd97ebb0baf58fcdc12ae4a"
Oct 30 01:45:59.738092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.737991    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cecb3050ac1f23091197f927ec5c3a22172c92b96bd97ebb0baf58fcdc12ae4a"} err="failed to get container status \"cecb3050ac1f23091197f927ec5c3a22172c92b96bd97ebb0baf58fcdc12ae4a\": rpc error: code = NotFound desc = could not find container \"cecb3050ac1f23091197f927ec5c3a22172c92b96bd97ebb0baf58fcdc12ae4a\": container with ID starting with cecb3050ac1f23091197f927ec5c3a22172c92b96bd97ebb0baf58fcdc12ae4a not found: ID does not exist"
Oct 30 01:45:59.773937 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.773905    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"]
Oct 30 01:45:59.779130 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.779107    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-73423b-openshift-default-85994d4b55-kbscf"]
Oct 30 01:45:59.795170 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:45:59.795144    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6897e74-0146-4ff4-89e3-4f1655fd8f35" path="/var/lib/kubelet/pods/b6897e74-0146-4ff4-89e3-4f1655fd8f35/volumes"
Oct 30 01:46:02.516298 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.516240    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:46:02.604325 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.604286    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-envoy\") pod \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") "
Oct 30 01:46:02.604531 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.604387    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-data\") pod \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") "
Oct 30 01:46:02.604531 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.604525    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-credential-socket\") pod \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") "
Oct 30 01:46:02.604672 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.604570    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-kube-api-access-jttlh\" (UniqueName: \"kubernetes.io/projected/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-kube-api-access-jttlh\") pod \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") "
Oct 30 01:46:02.604672 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.604623    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istiod-ca-cert\") pod \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") "
Oct 30 01:46:02.604785 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.604685    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-workload-socket\") pod \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") "
Oct 30 01:46:02.604860 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.604837    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-podinfo\") pod \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") "
Oct 30 01:46:02.604923 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.604907    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-workload-certs\") pod \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") "
Oct 30 01:46:02.604982 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.604945    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-token\" (UniqueName: \"kubernetes.io/projected/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-token\") pod \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\" (UID: \"31f94d41-d691-4cdc-a24b-fc976bc9e1c8\") "
Oct 30 01:46:02.606524 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.605536    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-workload-socket" pod "31f94d41-d691-4cdc-a24b-fc976bc9e1c8" (UID: "31f94d41-d691-4cdc-a24b-fc976bc9e1c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:46:02.606524 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.605701    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:46:02.606524 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.606127    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istiod-ca-cert" pod "31f94d41-d691-4cdc-a24b-fc976bc9e1c8" (UID: "31f94d41-d691-4cdc-a24b-fc976bc9e1c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:46:02.606524 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.606425    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-credential-socket" pod "31f94d41-d691-4cdc-a24b-fc976bc9e1c8" (UID: "31f94d41-d691-4cdc-a24b-fc976bc9e1c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:46:02.606524 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.606497    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-workload-certs" pod "31f94d41-d691-4cdc-a24b-fc976bc9e1c8" (UID: "31f94d41-d691-4cdc-a24b-fc976bc9e1c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:46:02.606976 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.606940    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-data" pod "31f94d41-d691-4cdc-a24b-fc976bc9e1c8" (UID: "31f94d41-d691-4cdc-a24b-fc976bc9e1c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:46:02.608405 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.608369    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-token" pod "31f94d41-d691-4cdc-a24b-fc976bc9e1c8" (UID: "31f94d41-d691-4cdc-a24b-fc976bc9e1c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:46:02.609368 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.609331    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-envoy" pod "31f94d41-d691-4cdc-a24b-fc976bc9e1c8" (UID: "31f94d41-d691-4cdc-a24b-fc976bc9e1c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:46:02.611033 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.611007    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-podinfo" pod "31f94d41-d691-4cdc-a24b-fc976bc9e1c8" (UID: "31f94d41-d691-4cdc-a24b-fc976bc9e1c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:46:02.611349 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.611317    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-kube-api-access-jttlh" pod "31f94d41-d691-4cdc-a24b-fc976bc9e1c8" (UID: "31f94d41-d691-4cdc-a24b-fc976bc9e1c8"). InnerVolumeSpecName "kube-api-access-jttlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:46:02.717134 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.715418    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:46:02.717134 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.715462    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:46:02.717134 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.715477    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:46:02.717134 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.715493    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jttlh\" (UniqueName: \"kubernetes.io/projected/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-kube-api-access-jttlh\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:46:02.717134 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.715508    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:46:02.717134 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.715522    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:46:02.717134 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.715534    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:46:02.717134 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.715549    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31f94d41-d691-4cdc-a24b-fc976bc9e1c8-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:46:02.739369 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.739338    4864 generic.go:356] "Generic (PLEG): container finished" podID="31f94d41-d691-4cdc-a24b-fc976bc9e1c8" containerID="86e6aa31c29cb1847eb0cff35f1e78063a74764aaa4b2c347a3bb959cc1d5498" exitCode=0
Oct 30 01:46:02.739604 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.739395    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"
Oct 30 01:46:02.739604 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.739420    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85" event={"ID":"31f94d41-d691-4cdc-a24b-fc976bc9e1c8","Type":"ContainerDied","Data":"86e6aa31c29cb1847eb0cff35f1e78063a74764aaa4b2c347a3bb959cc1d5498"}
Oct 30 01:46:02.739604 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.739524    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85" event={"ID":"31f94d41-d691-4cdc-a24b-fc976bc9e1c8","Type":"ContainerDied","Data":"e8e2314dfa13fa9282a40780852bc762ccee5f9ba19f8ae396af3e317d4c145c"}
Oct 30 01:46:02.739604 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.739555    4864 scope.go:122] "RemoveContainer" containerID="86e6aa31c29cb1847eb0cff35f1e78063a74764aaa4b2c347a3bb959cc1d5498"
Oct 30 01:46:02.751834 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.751816    4864 scope.go:122] "RemoveContainer" containerID="86e6aa31c29cb1847eb0cff35f1e78063a74764aaa4b2c347a3bb959cc1d5498"
Oct 30 01:46:02.752206 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:46:02.752182    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e6aa31c29cb1847eb0cff35f1e78063a74764aaa4b2c347a3bb959cc1d5498\": container with ID starting with 86e6aa31c29cb1847eb0cff35f1e78063a74764aaa4b2c347a3bb959cc1d5498 not found: ID does not exist" containerID="86e6aa31c29cb1847eb0cff35f1e78063a74764aaa4b2c347a3bb959cc1d5498"
Oct 30 01:46:02.752297 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.752215    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e6aa31c29cb1847eb0cff35f1e78063a74764aaa4b2c347a3bb959cc1d5498"} err="failed to get container status \"86e6aa31c29cb1847eb0cff35f1e78063a74764aaa4b2c347a3bb959cc1d5498\": rpc error: code = NotFound desc = could not find container \"86e6aa31c29cb1847eb0cff35f1e78063a74764aaa4b2c347a3bb959cc1d5498\": container with ID starting with 86e6aa31c29cb1847eb0cff35f1e78063a74764aaa4b2c347a3bb959cc1d5498 not found: ID does not exist"
Oct 30 01:46:02.789704 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.789621    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"]
Oct 30 01:46:02.794329 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:02.794296    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5457f2-openshift-default-69884bcb8c-jtv85"]
Oct 30 01:46:03.794382 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:03.794342    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31f94d41-d691-4cdc-a24b-fc976bc9e1c8" path="/var/lib/kubelet/pods/31f94d41-d691-4cdc-a24b-fc976bc9e1c8/volumes"
Oct 30 01:46:07.761897 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.761860    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"]
Oct 30 01:46:07.762777 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.762752    4864 cpu_manager.go:414] "containerMap: removing container" podUID="a18da34b-a9cb-4c2b-b7a3-b7921ba4425e" containerName="cleanup"
Oct 30 01:46:07.762869 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.762778    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a18da34b-a9cb-4c2b-b7a3-b7921ba4425e" containerName="cleanup"
Oct 30 01:46:07.762869 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.762793    4864 cpu_manager.go:414] "containerMap: removing container" podUID="a18da34b-a9cb-4c2b-b7a3-b7921ba4425e" containerName="cleanup"
Oct 30 01:46:07.762869 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.762801    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a18da34b-a9cb-4c2b-b7a3-b7921ba4425e" containerName="cleanup"
Oct 30 01:46:07.762869 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.762816    4864 cpu_manager.go:414] "containerMap: removing container" podUID="31f94d41-d691-4cdc-a24b-fc976bc9e1c8" containerName="istio-proxy"
Oct 30 01:46:07.762869 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.762824    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f94d41-d691-4cdc-a24b-fc976bc9e1c8" containerName="istio-proxy"
Oct 30 01:46:07.762869 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.762842    4864 cpu_manager.go:414] "containerMap: removing container" podUID="ce4812b0-4108-4d51-98cf-607aab4eebc6" containerName="cleanup"
Oct 30 01:46:07.762869 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.762850    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce4812b0-4108-4d51-98cf-607aab4eebc6" containerName="cleanup"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.762870    4864 cpu_manager.go:414] "containerMap: removing container" podUID="92a9c763-b5f0-4932-aa83-83b5afef6de9" containerName="payload-processing"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.762878    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a9c763-b5f0-4932-aa83-83b5afef6de9" containerName="payload-processing"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.762888    4864 cpu_manager.go:414] "containerMap: removing container" podUID="ce4812b0-4108-4d51-98cf-607aab4eebc6" containerName="cleanup"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.762895    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce4812b0-4108-4d51-98cf-607aab4eebc6" containerName="cleanup"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.762911    4864 cpu_manager.go:414] "containerMap: removing container" podUID="ba90b336-84e0-41ce-a673-29892a67100e" containerName="cleanup"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.762919    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba90b336-84e0-41ce-a673-29892a67100e" containerName="cleanup"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.762928    4864 cpu_manager.go:414] "containerMap: removing container" podUID="d9774ce4-b064-4823-a4dc-e915ff4820f0" containerName="cleanup"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.762934    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9774ce4-b064-4823-a4dc-e915ff4820f0" containerName="cleanup"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.762948    4864 cpu_manager.go:414] "containerMap: removing container" podUID="946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e" containerName="cleanup"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.762955    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e" containerName="cleanup"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.762967    4864 cpu_manager.go:414] "containerMap: removing container" podUID="241af13e-1e3a-466a-88b4-111de262ca2a" containerName="cleanup"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.762973    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="241af13e-1e3a-466a-88b4-111de262ca2a" containerName="cleanup"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.762993    4864 cpu_manager.go:414] "containerMap: removing container" podUID="e9245410-0ba2-4d32-abd2-cfa9e692fd78" containerName="cleanup"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.762999    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9245410-0ba2-4d32-abd2-cfa9e692fd78" containerName="cleanup"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763011    4864 cpu_manager.go:414] "containerMap: removing container" podUID="b6897e74-0146-4ff4-89e3-4f1655fd8f35" containerName="istio-proxy"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763018    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6897e74-0146-4ff4-89e3-4f1655fd8f35" containerName="istio-proxy"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763029    4864 cpu_manager.go:414] "containerMap: removing container" podUID="b6897e74-0146-4ff4-89e3-4f1655fd8f35" containerName="istio-proxy"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763036    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6897e74-0146-4ff4-89e3-4f1655fd8f35" containerName="istio-proxy"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763043    4864 cpu_manager.go:414] "containerMap: removing container" podUID="241af13e-1e3a-466a-88b4-111de262ca2a" containerName="cleanup"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763049    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="241af13e-1e3a-466a-88b4-111de262ca2a" containerName="cleanup"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763065    4864 cpu_manager.go:414] "containerMap: removing container" podUID="e9245410-0ba2-4d32-abd2-cfa9e692fd78" containerName="cleanup"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763092    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9245410-0ba2-4d32-abd2-cfa9e692fd78" containerName="cleanup"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763101    4864 cpu_manager.go:414] "containerMap: removing container" podUID="ba90b336-84e0-41ce-a673-29892a67100e" containerName="cleanup"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763108    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba90b336-84e0-41ce-a673-29892a67100e" containerName="cleanup"
Oct 30 01:46:07.763118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763123    4864 cpu_manager.go:414] "containerMap: removing container" podUID="241af13e-1e3a-466a-88b4-111de262ca2a" containerName="cleanup"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763130    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="241af13e-1e3a-466a-88b4-111de262ca2a" containerName="cleanup"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763145    4864 cpu_manager.go:414] "containerMap: removing container" podUID="d9774ce4-b064-4823-a4dc-e915ff4820f0" containerName="cleanup"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763152    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9774ce4-b064-4823-a4dc-e915ff4820f0" containerName="cleanup"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763162    4864 cpu_manager.go:414] "containerMap: removing container" podUID="e167ecf0-dd04-4531-9fbb-d985e9fe5132" containerName="payload-pre-processing"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763169    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e167ecf0-dd04-4531-9fbb-d985e9fe5132" containerName="payload-pre-processing"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763180    4864 cpu_manager.go:414] "containerMap: removing container" podUID="e9245410-0ba2-4d32-abd2-cfa9e692fd78" containerName="cleanup"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763187    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9245410-0ba2-4d32-abd2-cfa9e692fd78" containerName="cleanup"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763195    4864 cpu_manager.go:414] "containerMap: removing container" podUID="946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e" containerName="cleanup"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763202    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e" containerName="cleanup"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763368    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9245410-0ba2-4d32-abd2-cfa9e692fd78" containerName="cleanup"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763382    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba90b336-84e0-41ce-a673-29892a67100e" containerName="cleanup"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763393    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="241af13e-1e3a-466a-88b4-111de262ca2a" containerName="cleanup"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763404    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e" containerName="cleanup"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763413    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9774ce4-b064-4823-a4dc-e915ff4820f0" containerName="cleanup"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763423    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9774ce4-b064-4823-a4dc-e915ff4820f0" containerName="cleanup"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763431    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba90b336-84e0-41ce-a673-29892a67100e" containerName="cleanup"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763439    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="e167ecf0-dd04-4531-9fbb-d985e9fe5132" containerName="payload-pre-processing"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763452    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9245410-0ba2-4d32-abd2-cfa9e692fd78" containerName="cleanup"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763460    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e" containerName="cleanup"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763470    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="a18da34b-a9cb-4c2b-b7a3-b7921ba4425e" containerName="cleanup"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763480    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="a18da34b-a9cb-4c2b-b7a3-b7921ba4425e" containerName="cleanup"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763489    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="31f94d41-d691-4cdc-a24b-fc976bc9e1c8" containerName="istio-proxy"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763498    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce4812b0-4108-4d51-98cf-607aab4eebc6" containerName="cleanup"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763509    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="a18da34b-a9cb-4c2b-b7a3-b7921ba4425e" containerName="cleanup"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763517    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a9c763-b5f0-4932-aa83-83b5afef6de9" containerName="payload-processing"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763525    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce4812b0-4108-4d51-98cf-607aab4eebc6" containerName="cleanup"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763538    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6897e74-0146-4ff4-89e3-4f1655fd8f35" containerName="istio-proxy"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763547    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6897e74-0146-4ff4-89e3-4f1655fd8f35" containerName="istio-proxy"
Oct 30 01:46:07.763872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.763555    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="241af13e-1e3a-466a-88b4-111de262ca2a" containerName="cleanup"
Oct 30 01:46:07.767662 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.767642    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.781855 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.781829    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pjbq\" (UniqueName: \"kubernetes.io/projected/89b1105d-0395-4e80-a63b-7e8e0cce276d-kube-api-access-5pjbq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.781952 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.781884    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.781993 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.781964    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.782033 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.781992    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.782033 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.782024    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.782225 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.782204    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.782261 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.782237    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.782313 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.782295    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.782361 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.782350    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b1105d-0395-4e80-a63b-7e8e0cce276d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.796253 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.796223    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"]
Oct 30 01:46:07.883566 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.883531    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.883780 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.883597    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.883780 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.883657    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.883780 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.883714    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.884043 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.883800    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.884043 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.883827    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.884043 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.883871    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.884043 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.883935    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b1105d-0395-4e80-a63b-7e8e0cce276d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.884043 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.884015    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pjbq\" (UniqueName: \"kubernetes.io/projected/89b1105d-0395-4e80-a63b-7e8e0cce276d-kube-api-access-5pjbq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.884346 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.884058    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.884346 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.884131    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.884346 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.884193    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.884346 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.884277    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.885010 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.884980    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b1105d-0395-4e80-a63b-7e8e0cce276d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.886193 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.886174    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.886432 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.886415    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.906447 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.906414    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pjbq\" (UniqueName: \"kubernetes.io/projected/89b1105d-0395-4e80-a63b-7e8e0cce276d-kube-api-access-5pjbq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:07.906447 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:07.906436    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vwf7t\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:08.082038 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.081954    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:08.398907 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.398878    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"]
Oct 30 01:46:08.400726 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:46:08.400698    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b1105d_0395_4e80_a63b_7e8e0cce276d.slice/crio-11afd25fb8bd86920a28f43aeb5c7d0f094adeb8671e058b8e87c979ee966b08 WatchSource:0}: Error finding container 11afd25fb8bd86920a28f43aeb5c7d0f094adeb8671e058b8e87c979ee966b08: Status 404 returned error can't find the container with id 11afd25fb8bd86920a28f43aeb5c7d0f094adeb8671e058b8e87c979ee966b08
Oct 30 01:46:08.403510 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.403481    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:46:08.403584 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.403545    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:46:08.403584 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.403575    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:46:08.782755 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.782715    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t" event={"ID":"89b1105d-0395-4e80-a63b-7e8e0cce276d","Type":"ContainerStarted","Data":"ca24b5ad03af3279d18cb6f628f6947a3928918cf5028902146eaa2e6c999ce1"}
Oct 30 01:46:08.783267 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.782780    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t" event={"ID":"89b1105d-0395-4e80-a63b-7e8e0cce276d","Type":"ContainerStarted","Data":"11afd25fb8bd86920a28f43aeb5c7d0f094adeb8671e058b8e87c979ee966b08"}
Oct 30 01:46:08.825632 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.825579    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t" podStartSLOduration=1.825562225 podStartE2EDuration="1.825562225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:08.823410431 +0000 UTC m=+3881.161512340" watchObservedRunningTime="2026-10-30 01:46:08.825562225 +0000 UTC m=+3881.163664135"
Oct 30 01:46:08.957330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.957299    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"]
Oct 30 01:46:08.958092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.958053    4864 cpu_manager.go:414] "containerMap: removing container" podUID="d9774ce4-b064-4823-a4dc-e915ff4820f0" containerName="cleanup"
Oct 30 01:46:08.958092 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.958093    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9774ce4-b064-4823-a4dc-e915ff4820f0" containerName="cleanup"
Oct 30 01:46:08.958398 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.958115    4864 cpu_manager.go:414] "containerMap: removing container" podUID="946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e" containerName="cleanup"
Oct 30 01:46:08.958398 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.958123    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e" containerName="cleanup"
Oct 30 01:46:08.958398 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.958150    4864 cpu_manager.go:414] "containerMap: removing container" podUID="ba90b336-84e0-41ce-a673-29892a67100e" containerName="cleanup"
Oct 30 01:46:08.958398 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.958155    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba90b336-84e0-41ce-a673-29892a67100e" containerName="cleanup"
Oct 30 01:46:08.958398 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.958179    4864 cpu_manager.go:414] "containerMap: removing container" podUID="a18da34b-a9cb-4c2b-b7a3-b7921ba4425e" containerName="cleanup"
Oct 30 01:46:08.958398 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.958184    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a18da34b-a9cb-4c2b-b7a3-b7921ba4425e" containerName="cleanup"
Oct 30 01:46:08.958398 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.958224    4864 cpu_manager.go:414] "containerMap: removing container" podUID="ce4812b0-4108-4d51-98cf-607aab4eebc6" containerName="cleanup"
Oct 30 01:46:08.958398 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.958230    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce4812b0-4108-4d51-98cf-607aab4eebc6" containerName="cleanup"
Oct 30 01:46:08.958398 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.958343    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce4812b0-4108-4d51-98cf-607aab4eebc6" containerName="cleanup"
Oct 30 01:46:08.958398 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.958359    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9245410-0ba2-4d32-abd2-cfa9e692fd78" containerName="cleanup"
Oct 30 01:46:08.958398 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.958367    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="241af13e-1e3a-466a-88b4-111de262ca2a" containerName="cleanup"
Oct 30 01:46:08.958398 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.958374    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9774ce4-b064-4823-a4dc-e915ff4820f0" containerName="cleanup"
Oct 30 01:46:08.958398 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.958380    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba90b336-84e0-41ce-a673-29892a67100e" containerName="cleanup"
Oct 30 01:46:08.958398 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.958386    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="946d9b3b-d10a-4ccd-ae80-ffe5c421ff2e" containerName="cleanup"
Oct 30 01:46:08.961978 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.961960    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:08.976760 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.976733    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e6fc524e-openshift-default-dockercfg-sx6cz\""
Oct 30 01:46:08.988660 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:08.988631    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"]
Oct 30 01:46:09.005979 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.005945    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istiod-ca-cert\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.006331 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.006301    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-podinfo\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.006463 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.006379    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-data\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.006463 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.006454    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-workload-socket\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.006568 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.006531    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn48b\" (UniqueName: \"kubernetes.io/projected/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-kube-api-access-vn48b\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.006631 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.006589    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-token\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.006631 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.006616    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-envoy\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.006741 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.006654    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-workload-certs\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.006741 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.006703    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-credential-socket\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.104142 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.103595    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:09.104142 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.103751    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vwf7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 01:46:09.104142 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.103793    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t" podUID="89b1105d-0395-4e80-a63b-7e8e0cce276d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 01:46:09.121135 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.112954    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-token\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.121135 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.113009    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-envoy\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.121135 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.113056    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-workload-certs\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.121135 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.113131    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-credential-socket\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.121135 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.113198    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istiod-ca-cert\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.121135 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.113254    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-podinfo\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.121135 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.113327    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-data\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.121135 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.113422    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-workload-socket\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.121135 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.113511    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn48b\" (UniqueName: \"kubernetes.io/projected/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-kube-api-access-vn48b\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.140096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.132641    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istiod-ca-cert\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.140096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.132935    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-workload-certs\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.140096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.133219    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-credential-socket\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.140096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.133483    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-data\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.172123 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.141911    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-workload-socket\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.180153 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.152552    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-envoy\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.180375 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.161711    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-podinfo\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.195097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.193099    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn48b\" (UniqueName: \"kubernetes.io/projected/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-kube-api-access-vn48b\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.223095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.222676    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-token\") pod \"e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") " pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.288601 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.287703    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:09.626023 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.625994    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"]
Oct 30 01:46:09.627513 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:46:09.627481    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf683790c_2caa_4e0f_8b1a_f72ebd1ab40f.slice/crio-2d601e6c1071c41bb8207b393d1e7c4c01de11fb1bbd548a516d5edf7a8232fd WatchSource:0}: Error finding container 2d601e6c1071c41bb8207b393d1e7c4c01de11fb1bbd548a516d5edf7a8232fd: Status 404 returned error can't find the container with id 2d601e6c1071c41bb8207b393d1e7c4c01de11fb1bbd548a516d5edf7a8232fd
Oct 30 01:46:09.630419 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.630388    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:46:09.630501 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.630454    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:46:09.630501 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.630493    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:46:09.809634 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.802741    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2" event={"ID":"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f","Type":"ContainerStarted","Data":"797f988c5df9994ff609d35ce471fca2e4dbbbc736c08ca1e5f8d1fa7d89bd7c"}
Oct 30 01:46:09.809634 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.802815    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2" event={"ID":"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f","Type":"ContainerStarted","Data":"2d601e6c1071c41bb8207b393d1e7c4c01de11fb1bbd548a516d5edf7a8232fd"}
Oct 30 01:46:09.840049 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:09.839998    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2" podStartSLOduration=1.839985302 podStartE2EDuration="1.839985302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:09.838586666 +0000 UTC m=+3882.176688570" watchObservedRunningTime="2026-10-30 01:46:09.839985302 +0000 UTC m=+3882.178087211"
Oct 30 01:46:10.088140 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:10.087397    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vwf7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 01:46:10.088140 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:10.087449    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t" podUID="89b1105d-0395-4e80-a63b-7e8e0cce276d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 01:46:10.291517 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:10.291486    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:10.298064 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:10.298041    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:10.803085 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:10.803041    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:10.803630 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:10.803607    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:46:11.082729 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:11.082488    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vwf7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 01:46:11.082729 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:11.082547    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t" podUID="89b1105d-0395-4e80-a63b-7e8e0cce276d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 01:46:12.082798 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:12.082745    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vwf7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 01:46:12.083238 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:12.082807    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t" podUID="89b1105d-0395-4e80-a63b-7e8e0cce276d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 01:46:13.082540 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:13.082500    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vwf7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 01:46:13.082713 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:13.082564    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t" podUID="89b1105d-0395-4e80-a63b-7e8e0cce276d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 01:46:14.082767 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:14.082728    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vwf7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 01:46:14.083227 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:14.082794    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t" podUID="89b1105d-0395-4e80-a63b-7e8e0cce276d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 01:46:15.082654 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:15.082614    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vwf7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 01:46:15.082843 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:15.082679    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t" podUID="89b1105d-0395-4e80-a63b-7e8e0cce276d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 01:46:16.082602 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:16.082563    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vwf7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 01:46:16.082845 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:16.082641    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t" podUID="89b1105d-0395-4e80-a63b-7e8e0cce276d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 01:46:17.898095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:17.897162    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6hcmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:46:17.898095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:17.897239    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx" podUID="bc75808d-fb11-4820-939a-18037b0c2791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:46:18.095493 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:18.095156    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-89zld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:46:18.095493 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:18.095221    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld" podUID="9de67abe-4f7c-441f-9e3e-cf34c74068d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:46:18.145225 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:18.145181    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vwf7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:46:18.145437 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:18.145247    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t" podUID="89b1105d-0395-4e80-a63b-7e8e0cce276d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:46:18.508110 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:18.508062    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:18.508733 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:18.508702    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:18.509920 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:18.509895    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:46:19.043199 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:46:19.043152    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:46:22.002921 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:22.002887    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc"]
Oct 30 01:46:22.036387 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:22.036354    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6"]
Oct 30 01:46:27.004584 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:27.004541    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc" podUID="3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48" containerName="payload-processing" containerID="cri-o://63b7117079445d8d12e30dd3e6a5a644210a14fb03848cc7b053de2da9317005" gracePeriod=25
Oct 30 01:46:27.036737 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:27.036704    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6" podUID="e3d5cee2-a8bf-4694-b410-48282432dc90" containerName="payload-pre-processing" containerID="cri-o://0fe642ce29d744c4da369343d0eb447051b19cdcac27d9c536bab09ec8068ba8" gracePeriod=25
Oct 30 01:46:28.603609 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:28.603580    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:46:28.624241 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:28.624212    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:46:36.352503 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:36.352466    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6" podUID="e3d5cee2-a8bf-4694-b410-48282432dc90" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:46:36.352503 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:36.352485    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc" podUID="3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:46:42.107877 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:42.107649    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"]
Oct 30 01:46:42.108415 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:42.107938    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs" podUID="2d617aa1-07b4-400c-8a69-f356e0ae7500" containerName="istio-proxy" containerID="cri-o://ec65824927ad61dfc61622762eae2b759b787abe670700a977d31d6fd8f7bab3" gracePeriod=30
Oct 30 01:46:46.352902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:46.352866    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6" podUID="e3d5cee2-a8bf-4694-b410-48282432dc90" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:46:46.353337 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:46.352917    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc" podUID="3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:46:47.198176 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.198143    4864 generic.go:356] "Generic (PLEG): container finished" podID="3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48" containerID="63b7117079445d8d12e30dd3e6a5a644210a14fb03848cc7b053de2da9317005" exitCode=0
Oct 30 01:46:47.198373 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.198320    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc" event={"ID":"3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48","Type":"ContainerDied","Data":"63b7117079445d8d12e30dd3e6a5a644210a14fb03848cc7b053de2da9317005"}
Oct 30 01:46:47.200016 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.199985    4864 generic.go:356] "Generic (PLEG): container finished" podID="e3d5cee2-a8bf-4694-b410-48282432dc90" containerID="0fe642ce29d744c4da369343d0eb447051b19cdcac27d9c536bab09ec8068ba8" exitCode=0
Oct 30 01:46:47.200187 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.200111    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6" event={"ID":"e3d5cee2-a8bf-4694-b410-48282432dc90","Type":"ContainerDied","Data":"0fe642ce29d744c4da369343d0eb447051b19cdcac27d9c536bab09ec8068ba8"}
Oct 30 01:46:47.327859 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.327828    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc"
Oct 30 01:46:47.376486 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.376452    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48-kube-api-access-n8lj9\" (UniqueName: \"kubernetes.io/projected/3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48-kube-api-access-n8lj9\") pod \"3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48\" (UID: \"3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48\") "
Oct 30 01:46:47.376922 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.376679    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48-plugins-config-volume\") pod \"3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48\" (UID: \"3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48\") "
Oct 30 01:46:47.378340 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.378306    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48-plugins-config-volume" pod "3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48" (UID: "3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:46:47.381269 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.381240    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48-kube-api-access-n8lj9" pod "3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48" (UID: "3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48"). InnerVolumeSpecName "kube-api-access-n8lj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:46:47.409310 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.409280    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6"
Oct 30 01:46:47.446507 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.446483    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:46:47.477950 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.477920    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d617aa1-07b4-400c-8a69-f356e0ae7500-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d617aa1-07b4-400c-8a69-f356e0ae7500-istiod-ca-cert\") pod \"2d617aa1-07b4-400c-8a69-f356e0ae7500\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") "
Oct 30 01:46:47.478130 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.477983    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-envoy\") pod \"2d617aa1-07b4-400c-8a69-f356e0ae7500\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") "
Oct 30 01:46:47.478130 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.478055    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3d5cee2-a8bf-4694-b410-48282432dc90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3d5cee2-a8bf-4694-b410-48282432dc90-plugins-config-volume\") pod \"e3d5cee2-a8bf-4694-b410-48282432dc90\" (UID: \"e3d5cee2-a8bf-4694-b410-48282432dc90\") "
Oct 30 01:46:47.478130 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.478105    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-workload-socket\") pod \"2d617aa1-07b4-400c-8a69-f356e0ae7500\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") "
Oct 30 01:46:47.478331 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.478183    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-podinfo\") pod \"2d617aa1-07b4-400c-8a69-f356e0ae7500\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") "
Oct 30 01:46:47.478331 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.478236    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3d5cee2-a8bf-4694-b410-48282432dc90-kube-api-access-mngdz\" (UniqueName: \"kubernetes.io/projected/e3d5cee2-a8bf-4694-b410-48282432dc90-kube-api-access-mngdz\") pod \"e3d5cee2-a8bf-4694-b410-48282432dc90\" (UID: \"e3d5cee2-a8bf-4694-b410-48282432dc90\") "
Oct 30 01:46:47.478331 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.478267    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-credential-socket\") pod \"2d617aa1-07b4-400c-8a69-f356e0ae7500\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") "
Oct 30 01:46:47.478498 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.478350    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d617aa1-07b4-400c-8a69-f356e0ae7500-kube-api-access-gwq7j\" (UniqueName: \"kubernetes.io/projected/2d617aa1-07b4-400c-8a69-f356e0ae7500-kube-api-access-gwq7j\") pod \"2d617aa1-07b4-400c-8a69-f356e0ae7500\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") "
Oct 30 01:46:47.478498 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.478374    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-token\" (UniqueName: \"kubernetes.io/projected/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-token\") pod \"2d617aa1-07b4-400c-8a69-f356e0ae7500\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") "
Oct 30 01:46:47.478498 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.478436    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d617aa1-07b4-400c-8a69-f356e0ae7500-istiod-ca-cert" pod "2d617aa1-07b4-400c-8a69-f356e0ae7500" (UID: "2d617aa1-07b4-400c-8a69-f356e0ae7500"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:46:47.478659 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.478522    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-data\") pod \"2d617aa1-07b4-400c-8a69-f356e0ae7500\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") "
Oct 30 01:46:47.478659 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.478529    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3d5cee2-a8bf-4694-b410-48282432dc90-plugins-config-volume" pod "e3d5cee2-a8bf-4694-b410-48282432dc90" (UID: "e3d5cee2-a8bf-4694-b410-48282432dc90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:46:47.478659 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.478606    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-workload-certs\") pod \"2d617aa1-07b4-400c-8a69-f356e0ae7500\" (UID: \"2d617aa1-07b4-400c-8a69-f356e0ae7500\") "
Oct 30 01:46:47.478812 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.478790    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-workload-socket" pod "2d617aa1-07b4-400c-8a69-f356e0ae7500" (UID: "2d617aa1-07b4-400c-8a69-f356e0ae7500"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:46:47.478946 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.478924    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-credential-socket" pod "2d617aa1-07b4-400c-8a69-f356e0ae7500" (UID: "2d617aa1-07b4-400c-8a69-f356e0ae7500"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:46:47.479125 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.479104    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-data" pod "2d617aa1-07b4-400c-8a69-f356e0ae7500" (UID: "2d617aa1-07b4-400c-8a69-f356e0ae7500"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:46:47.479225 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.479201    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-workload-certs" pod "2d617aa1-07b4-400c-8a69-f356e0ae7500" (UID: "2d617aa1-07b4-400c-8a69-f356e0ae7500"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:46:47.479446 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.479429    4864 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3d5cee2-a8bf-4694-b410-48282432dc90-plugins-config-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:46:47.479513 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.479450    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:46:47.479513 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.479464    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:46:47.479513 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.479473    4864 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48-plugins-config-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:46:47.479513 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.479482    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:46:47.479513 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.479491    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:46:47.479513 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.479498    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8lj9\" (UniqueName: \"kubernetes.io/projected/3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48-kube-api-access-n8lj9\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:46:47.479513 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.479511    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d617aa1-07b4-400c-8a69-f356e0ae7500-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:46:47.480719 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.480682    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d5cee2-a8bf-4694-b410-48282432dc90-kube-api-access-mngdz" pod "e3d5cee2-a8bf-4694-b410-48282432dc90" (UID: "e3d5cee2-a8bf-4694-b410-48282432dc90"). InnerVolumeSpecName "kube-api-access-mngdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:46:47.481223 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.481198    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d617aa1-07b4-400c-8a69-f356e0ae7500-kube-api-access-gwq7j" pod "2d617aa1-07b4-400c-8a69-f356e0ae7500" (UID: "2d617aa1-07b4-400c-8a69-f356e0ae7500"). InnerVolumeSpecName "kube-api-access-gwq7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:46:47.481444 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.481424    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-envoy" pod "2d617aa1-07b4-400c-8a69-f356e0ae7500" (UID: "2d617aa1-07b4-400c-8a69-f356e0ae7500"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:46:47.482033 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.482012    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-podinfo" pod "2d617aa1-07b4-400c-8a69-f356e0ae7500" (UID: "2d617aa1-07b4-400c-8a69-f356e0ae7500"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:46:47.483116 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.483099    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-token" pod "2d617aa1-07b4-400c-8a69-f356e0ae7500" (UID: "2d617aa1-07b4-400c-8a69-f356e0ae7500"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:46:47.580692 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.580607    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:46:47.580692 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.580640    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mngdz\" (UniqueName: \"kubernetes.io/projected/e3d5cee2-a8bf-4694-b410-48282432dc90-kube-api-access-mngdz\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:46:47.580692 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.580651    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwq7j\" (UniqueName: \"kubernetes.io/projected/2d617aa1-07b4-400c-8a69-f356e0ae7500-kube-api-access-gwq7j\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:46:47.580692 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.580661    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:46:47.580692 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:47.580670    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d617aa1-07b4-400c-8a69-f356e0ae7500-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:46:48.208227 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:48.208193    4864 generic.go:356] "Generic (PLEG): container finished" podID="2d617aa1-07b4-400c-8a69-f356e0ae7500" containerID="ec65824927ad61dfc61622762eae2b759b787abe670700a977d31d6fd8f7bab3" exitCode=0
Oct 30 01:46:48.208423 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:48.208247    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"
Oct 30 01:46:48.208423 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:48.208279    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs" event={"ID":"2d617aa1-07b4-400c-8a69-f356e0ae7500","Type":"ContainerDied","Data":"ec65824927ad61dfc61622762eae2b759b787abe670700a977d31d6fd8f7bab3"}
Oct 30 01:46:48.208423 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:48.208332    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs" event={"ID":"2d617aa1-07b4-400c-8a69-f356e0ae7500","Type":"ContainerDied","Data":"cbf884e74d8abfa75f98f98ec424cd0ead827e1a1cace81a435559cfb5cf950b"}
Oct 30 01:46:48.208423 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:48.208359    4864 scope.go:122] "RemoveContainer" containerID="ec65824927ad61dfc61622762eae2b759b787abe670700a977d31d6fd8f7bab3"
Oct 30 01:46:48.209924 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:48.209904    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6"
Oct 30 01:46:48.210052 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:48.209938    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6" event={"ID":"e3d5cee2-a8bf-4694-b410-48282432dc90","Type":"ContainerDied","Data":"ae48367ff11b26ee40dce95c84bc1dca7a02ba06a3207c82b2cf5e1527c4da1e"}
Oct 30 01:46:48.211674 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:48.211651    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc" event={"ID":"3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48","Type":"ContainerDied","Data":"98cc2a530ac70395dda5cf02a359307aa7f550e00265e1db57dca87c2cdbfdba"}
Oct 30 01:46:48.211800 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:48.211674    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc"
Oct 30 01:46:48.219677 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:48.219660    4864 scope.go:122] "RemoveContainer" containerID="ec65824927ad61dfc61622762eae2b759b787abe670700a977d31d6fd8f7bab3"
Oct 30 01:46:48.219993 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:46:48.219963    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec65824927ad61dfc61622762eae2b759b787abe670700a977d31d6fd8f7bab3\": container with ID starting with ec65824927ad61dfc61622762eae2b759b787abe670700a977d31d6fd8f7bab3 not found: ID does not exist" containerID="ec65824927ad61dfc61622762eae2b759b787abe670700a977d31d6fd8f7bab3"
Oct 30 01:46:48.220111 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:48.220001    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec65824927ad61dfc61622762eae2b759b787abe670700a977d31d6fd8f7bab3"} err="failed to get container status \"ec65824927ad61dfc61622762eae2b759b787abe670700a977d31d6fd8f7bab3\": rpc error: code = NotFound desc = could not find container \"ec65824927ad61dfc61622762eae2b759b787abe670700a977d31d6fd8f7bab3\": container with ID starting with ec65824927ad61dfc61622762eae2b759b787abe670700a977d31d6fd8f7bab3 not found: ID does not exist"
Oct 30 01:46:48.220111 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:48.220026    4864 scope.go:122] "RemoveContainer" containerID="0fe642ce29d744c4da369343d0eb447051b19cdcac27d9c536bab09ec8068ba8"
Oct 30 01:46:48.229730 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:48.229705    4864 scope.go:122] "RemoveContainer" containerID="63b7117079445d8d12e30dd3e6a5a644210a14fb03848cc7b053de2da9317005"
Oct 30 01:46:48.276986 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:48.276961    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6"]
Oct 30 01:46:48.282248 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:48.282225    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6aa9aa-9655fcfff-jshw6"]
Oct 30 01:46:48.321904 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:48.321826    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc"]
Oct 30 01:46:48.325064 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:48.325039    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6aa9aa-58d57d6cd6-w5cwc"]
Oct 30 01:46:48.354661 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:48.354624    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"]
Oct 30 01:46:48.359606 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:48.359583    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6aa9aa-openshift-default-8d7c8d5d9-n79vs"]
Oct 30 01:46:49.040584 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:46:49.040542    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:46:49.794756 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:49.794719    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d617aa1-07b4-400c-8a69-f356e0ae7500" path="/var/lib/kubelet/pods/2d617aa1-07b4-400c-8a69-f356e0ae7500/volumes"
Oct 30 01:46:49.795195 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:49.795180    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48" path="/var/lib/kubelet/pods/3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48/volumes"
Oct 30 01:46:49.795563 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:49.795549    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3d5cee2-a8bf-4694-b410-48282432dc90" path="/var/lib/kubelet/pods/e3d5cee2-a8bf-4694-b410-48282432dc90/volumes"
Oct 30 01:46:53.985112 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:46:53.983145    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29" podUID="e0ab9bcb-f48a-404b-b787-9f8c43b58722" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:47:19.041425 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:47:19.041388    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:47:30.533449 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:30.533412    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7"]
Oct 30 01:47:30.533902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:30.533619    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7" podUID="9254f264-647e-42f6-876a-3b38efcd829f" containerName="maas-api" containerID="cri-o://d515627ca85a88eb50ceb7e92fa43e4bb6f91b557dae2678b26cbc75e0d32f1e" gracePeriod=30
Oct 30 01:47:30.891466 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:30.891434    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7"
Oct 30 01:47:30.912330 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:30.912299    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9254f264-647e-42f6-876a-3b38efcd829f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9254f264-647e-42f6-876a-3b38efcd829f-openshift-service-ca\") pod \"9254f264-647e-42f6-876a-3b38efcd829f\" (UID: \"9254f264-647e-42f6-876a-3b38efcd829f\") "
Oct 30 01:47:30.912521 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:30.912389    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9254f264-647e-42f6-876a-3b38efcd829f-kube-api-access-dr5lw\" (UniqueName: \"kubernetes.io/projected/9254f264-647e-42f6-876a-3b38efcd829f-kube-api-access-dr5lw\") pod \"9254f264-647e-42f6-876a-3b38efcd829f\" (UID: \"9254f264-647e-42f6-876a-3b38efcd829f\") "
Oct 30 01:47:30.912597 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:30.912526    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9254f264-647e-42f6-876a-3b38efcd829f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9254f264-647e-42f6-876a-3b38efcd829f-metrics-certs\") pod \"9254f264-647e-42f6-876a-3b38efcd829f\" (UID: \"9254f264-647e-42f6-876a-3b38efcd829f\") "
Oct 30 01:47:30.912597 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:30.912571    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9254f264-647e-42f6-876a-3b38efcd829f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9254f264-647e-42f6-876a-3b38efcd829f-maas-api-tls\") pod \"9254f264-647e-42f6-876a-3b38efcd829f\" (UID: \"9254f264-647e-42f6-876a-3b38efcd829f\") "
Oct 30 01:47:30.912894 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:30.912694    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9254f264-647e-42f6-876a-3b38efcd829f-openshift-service-ca" pod "9254f264-647e-42f6-876a-3b38efcd829f" (UID: "9254f264-647e-42f6-876a-3b38efcd829f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:47:30.913212 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:30.913189    4864 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9254f264-647e-42f6-876a-3b38efcd829f-openshift-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:47:30.914728 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:30.914687    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9254f264-647e-42f6-876a-3b38efcd829f-kube-api-access-dr5lw" pod "9254f264-647e-42f6-876a-3b38efcd829f" (UID: "9254f264-647e-42f6-876a-3b38efcd829f"). InnerVolumeSpecName "kube-api-access-dr5lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:47:30.914840 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:30.914765    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9254f264-647e-42f6-876a-3b38efcd829f-metrics-certs" pod "9254f264-647e-42f6-876a-3b38efcd829f" (UID: "9254f264-647e-42f6-876a-3b38efcd829f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:47:30.914840 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:30.914790    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9254f264-647e-42f6-876a-3b38efcd829f-maas-api-tls" pod "9254f264-647e-42f6-876a-3b38efcd829f" (UID: "9254f264-647e-42f6-876a-3b38efcd829f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:47:31.015237 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:31.015137    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr5lw\" (UniqueName: \"kubernetes.io/projected/9254f264-647e-42f6-876a-3b38efcd829f-kube-api-access-dr5lw\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:47:31.015237 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:31.015176    4864 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9254f264-647e-42f6-876a-3b38efcd829f-metrics-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:47:31.015237 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:31.015191    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9254f264-647e-42f6-876a-3b38efcd829f-maas-api-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:47:31.588531 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:31.588484    4864 generic.go:356] "Generic (PLEG): container finished" podID="9254f264-647e-42f6-876a-3b38efcd829f" containerID="d515627ca85a88eb50ceb7e92fa43e4bb6f91b557dae2678b26cbc75e0d32f1e" exitCode=0
Oct 30 01:47:31.589035 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:31.588538    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7"
Oct 30 01:47:31.589035 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:31.588549    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7" event={"ID":"9254f264-647e-42f6-876a-3b38efcd829f","Type":"ContainerDied","Data":"d515627ca85a88eb50ceb7e92fa43e4bb6f91b557dae2678b26cbc75e0d32f1e"}
Oct 30 01:47:31.589035 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:31.588600    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7" event={"ID":"9254f264-647e-42f6-876a-3b38efcd829f","Type":"ContainerDied","Data":"db1e42e59fffb63ecd85d2406556999a43a40a9e7a3f074b22317e277d7d23f8"}
Oct 30 01:47:31.589035 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:31.588628    4864 scope.go:122] "RemoveContainer" containerID="d515627ca85a88eb50ceb7e92fa43e4bb6f91b557dae2678b26cbc75e0d32f1e"
Oct 30 01:47:31.599960 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:31.599945    4864 scope.go:122] "RemoveContainer" containerID="d515627ca85a88eb50ceb7e92fa43e4bb6f91b557dae2678b26cbc75e0d32f1e"
Oct 30 01:47:31.600245 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:47:31.600226    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d515627ca85a88eb50ceb7e92fa43e4bb6f91b557dae2678b26cbc75e0d32f1e\": container with ID starting with d515627ca85a88eb50ceb7e92fa43e4bb6f91b557dae2678b26cbc75e0d32f1e not found: ID does not exist" containerID="d515627ca85a88eb50ceb7e92fa43e4bb6f91b557dae2678b26cbc75e0d32f1e"
Oct 30 01:47:31.600289 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:31.600254    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d515627ca85a88eb50ceb7e92fa43e4bb6f91b557dae2678b26cbc75e0d32f1e"} err="failed to get container status \"d515627ca85a88eb50ceb7e92fa43e4bb6f91b557dae2678b26cbc75e0d32f1e\": rpc error: code = NotFound desc = could not find container \"d515627ca85a88eb50ceb7e92fa43e4bb6f91b557dae2678b26cbc75e0d32f1e\": container with ID starting with d515627ca85a88eb50ceb7e92fa43e4bb6f91b557dae2678b26cbc75e0d32f1e not found: ID does not exist"
Oct 30 01:47:31.643417 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:31.643386    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7"]
Oct 30 01:47:31.676350 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:31.676298    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1a4b17-5bf8bf4dc5-p6wd7"]
Oct 30 01:47:31.793836 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:31.793802    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9254f264-647e-42f6-876a-3b38efcd829f" path="/var/lib/kubelet/pods/9254f264-647e-42f6-876a-3b38efcd829f/volumes"
Oct 30 01:47:40.651159 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:40.651042    4864 generic.go:356] "Generic (PLEG): container finished" podID="f683790c-2caa-4e0f-8b1a-f72ebd1ab40f" containerID="797f988c5df9994ff609d35ce471fca2e4dbbbc736c08ca1e5f8d1fa7d89bd7c" exitCode=0
Oct 30 01:47:40.651159 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:40.651101    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2" event={"ID":"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f","Type":"ContainerDied","Data":"797f988c5df9994ff609d35ce471fca2e4dbbbc736c08ca1e5f8d1fa7d89bd7c"}
Oct 30 01:47:40.651753 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:40.651616    4864 scope.go:122] "RemoveContainer" containerID="797f988c5df9994ff609d35ce471fca2e4dbbbc736c08ca1e5f8d1fa7d89bd7c"
Oct 30 01:47:40.653451 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:40.653424    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:47:40.653526 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:40.653488    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:47:40.653526 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:40.653520    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:47:40.802846 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:40.802805    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:47:41.675096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:41.661960    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2" event={"ID":"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f","Type":"ContainerStarted","Data":"f2d3cc045e2d122d599fd0579dc14e66031bfb169227283ebc6d0e1536dadea1"}
Oct 30 01:47:42.290991 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:42.290956    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:47:42.292205 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:42.292177    4864 patch_prober.go:29] interesting pod/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 30 01:47:42.292354 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:42.292229    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2" podUID="f683790c-2caa-4e0f-8b1a-f72ebd1ab40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 30 01:47:43.290762 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:43.290724    4864 patch_prober.go:29] interesting pod/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 30 01:47:43.291166 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:43.290785    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2" podUID="f683790c-2caa-4e0f-8b1a-f72ebd1ab40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 30 01:47:44.291627 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:44.291591    4864 patch_prober.go:29] interesting pod/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 30 01:47:44.292053 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:44.291645    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2" podUID="f683790c-2caa-4e0f-8b1a-f72ebd1ab40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 30 01:47:45.291100 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:45.291033    4864 patch_prober.go:29] interesting pod/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 30 01:47:45.291322 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:45.291128    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2" podUID="f683790c-2caa-4e0f-8b1a-f72ebd1ab40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 30 01:47:46.291036 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:46.290998    4864 patch_prober.go:29] interesting pod/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 30 01:47:46.291465 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:46.291052    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2" podUID="f683790c-2caa-4e0f-8b1a-f72ebd1ab40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 30 01:47:47.296341 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:47.296309    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:47:47.296779 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:47.296760    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:47:47.297446 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:47:47.297423    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:47:49.043444 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:47:49.043412    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:48:11.109441 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.109340    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"]
Oct 30 01:48:11.110123 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.110099    4864 cpu_manager.go:414] "containerMap: removing container" podUID="e3d5cee2-a8bf-4694-b410-48282432dc90" containerName="payload-pre-processing"
Oct 30 01:48:11.110123 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.110122    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3d5cee2-a8bf-4694-b410-48282432dc90" containerName="payload-pre-processing"
Oct 30 01:48:11.110331 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.110150    4864 cpu_manager.go:414] "containerMap: removing container" podUID="9254f264-647e-42f6-876a-3b38efcd829f" containerName="maas-api"
Oct 30 01:48:11.110331 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.110155    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9254f264-647e-42f6-876a-3b38efcd829f" containerName="maas-api"
Oct 30 01:48:11.110331 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.110168    4864 cpu_manager.go:414] "containerMap: removing container" podUID="2d617aa1-07b4-400c-8a69-f356e0ae7500" containerName="istio-proxy"
Oct 30 01:48:11.110331 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.110174    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d617aa1-07b4-400c-8a69-f356e0ae7500" containerName="istio-proxy"
Oct 30 01:48:11.110331 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.110188    4864 cpu_manager.go:414] "containerMap: removing container" podUID="3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48" containerName="payload-processing"
Oct 30 01:48:11.110331 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.110193    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48" containerName="payload-processing"
Oct 30 01:48:11.110688 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.110350    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a0430bb-a6d4-4cad-8a3b-8a5f23ed2e48" containerName="payload-processing"
Oct 30 01:48:11.110688 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.110365    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3d5cee2-a8bf-4694-b410-48282432dc90" containerName="payload-pre-processing"
Oct 30 01:48:11.110688 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.110374    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="9254f264-647e-42f6-876a-3b38efcd829f" containerName="maas-api"
Oct 30 01:48:11.110688 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.110384    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d617aa1-07b4-400c-8a69-f356e0ae7500" containerName="istio-proxy"
Oct 30 01:48:11.113002 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.112980    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:48:11.137066 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.137030    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-8ae41e\"/\"kube-root-ca.crt\""
Oct 30 01:48:11.138594 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.138567    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-8ae41e\"/\"default-dockercfg-vbqhs\""
Oct 30 01:48:11.138594 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.138586    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-8ae41e\"/\"sub-test-model-8ae41e-kserve-self-signed-certs\""
Oct 30 01:48:11.141481 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.141454    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"]
Oct 30 01:48:11.143939 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.143914    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-tmp-dir\") pod \"sub-test-model-8ae41e-kserve-679b595c85-qkn25\" (UID: \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:48:11.144058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.143962    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft2fk\" (UniqueName: \"kubernetes.io/projected/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-kube-api-access-ft2fk\") pod \"sub-test-model-8ae41e-kserve-679b595c85-qkn25\" (UID: \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:48:11.144058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.143992    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-model-cache\") pod \"sub-test-model-8ae41e-kserve-679b595c85-qkn25\" (UID: \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:48:11.144230 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.144129    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-home\") pod \"sub-test-model-8ae41e-kserve-679b595c85-qkn25\" (UID: \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:48:11.144291 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.144263    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-tls-certs\") pod \"sub-test-model-8ae41e-kserve-679b595c85-qkn25\" (UID: \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:48:11.144398 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.144365    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-dshm\") pod \"sub-test-model-8ae41e-kserve-679b595c85-qkn25\" (UID: \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:48:11.226933 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.226900    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-8ae41e\"/\"openshift-service-ca.crt\""
Oct 30 01:48:11.245516 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.245481    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-tls-certs\") pod \"sub-test-model-8ae41e-kserve-679b595c85-qkn25\" (UID: \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:48:11.245686 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.245535    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-dshm\") pod \"sub-test-model-8ae41e-kserve-679b595c85-qkn25\" (UID: \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:48:11.245738 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.245690    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-tmp-dir\") pod \"sub-test-model-8ae41e-kserve-679b595c85-qkn25\" (UID: \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:48:11.245738 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.245720    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft2fk\" (UniqueName: \"kubernetes.io/projected/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-kube-api-access-ft2fk\") pod \"sub-test-model-8ae41e-kserve-679b595c85-qkn25\" (UID: \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:48:11.245843 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.245740    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-model-cache\") pod \"sub-test-model-8ae41e-kserve-679b595c85-qkn25\" (UID: \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:48:11.245843 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.245773    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-home\") pod \"sub-test-model-8ae41e-kserve-679b595c85-qkn25\" (UID: \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:48:11.246165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.246134    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-tmp-dir\") pod \"sub-test-model-8ae41e-kserve-679b595c85-qkn25\" (UID: \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:48:11.246165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.246156    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-model-cache\") pod \"sub-test-model-8ae41e-kserve-679b595c85-qkn25\" (UID: \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:48:11.246390 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.246296    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-home\") pod \"sub-test-model-8ae41e-kserve-679b595c85-qkn25\" (UID: \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:48:11.247965 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.247940    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-dshm\") pod \"sub-test-model-8ae41e-kserve-679b595c85-qkn25\" (UID: \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:48:11.248217 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.248199    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-tls-certs\") pod \"sub-test-model-8ae41e-kserve-679b595c85-qkn25\" (UID: \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:48:11.267545 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.267514    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft2fk\" (UniqueName: \"kubernetes.io/projected/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-kube-api-access-ft2fk\") pod \"sub-test-model-8ae41e-kserve-679b595c85-qkn25\" (UID: \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:48:11.431724 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.430450    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:48:11.763260 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:11.763227    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"]
Oct 30 01:48:11.766927 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:48:11.766898    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55dbdf6b_c1f2_4e0a_907a_7cd59c30e989.slice/crio-ce3b8957a1f0e44288ac0597d8575f4ca283417efe83a98d1316782533b3359d WatchSource:0}: Error finding container ce3b8957a1f0e44288ac0597d8575f4ca283417efe83a98d1316782533b3359d: Status 404 returned error can't find the container with id ce3b8957a1f0e44288ac0597d8575f4ca283417efe83a98d1316782533b3359d
Oct 30 01:48:12.013993 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:12.013904    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25" event={"ID":"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989","Type":"ContainerStarted","Data":"79e76a1a8f18ff1b40faa2390aceb61c12ff600ab53bbac4a3ccaa654239f6cf"}
Oct 30 01:48:12.013993 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:12.013954    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25" event={"ID":"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989","Type":"ContainerStarted","Data":"ce3b8957a1f0e44288ac0597d8575f4ca283417efe83a98d1316782533b3359d"}
Oct 30 01:48:12.054945 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:12.054893    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25" podStartSLOduration=1.054879013 podStartE2EDuration="1.054879013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:12.052046809 +0000 UTC m=+4004.390148742" watchObservedRunningTime="2026-10-30 01:48:12.054879013 +0000 UTC m=+4004.392980924"
Oct 30 01:48:12.431917 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:12.431825    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:48:19.075516 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:48:19.075482    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:48:21.431824 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:21.431734    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:48:21.444726 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:21.444702    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:48:22.094469 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:22.094439    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:48:40.769496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:40.769459    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"]
Oct 30 01:48:40.773653 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:40.773635    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:48:40.788932 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:40.788900    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-35042a\"/\"default-dockercfg-fpwtx\""
Oct 30 01:48:40.799620 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:40.799590    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-35042a\"/\"kube-root-ca.crt\""
Oct 30 01:48:40.799620 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:40.799594    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-35042a\"/\"openshift-service-ca.crt\""
Oct 30 01:48:40.805242 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:40.805212    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"]
Oct 30 01:48:40.880148 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:40.880119    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-35042a\"/\"rate-test-model-35042a-kserve-self-signed-certs\""
Oct 30 01:48:40.887863 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:40.887830    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-home\") pod \"rate-test-model-35042a-kserve-7f59b44fd5-vk4sj\" (UID: \"a87828aa-8038-4643-834a-bbdb78432874\") " pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:48:40.888010 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:40.887908    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a87828aa-8038-4643-834a-bbdb78432874-tls-certs\") pod \"rate-test-model-35042a-kserve-7f59b44fd5-vk4sj\" (UID: \"a87828aa-8038-4643-834a-bbdb78432874\") " pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:48:40.888246 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:40.888218    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-dshm\") pod \"rate-test-model-35042a-kserve-7f59b44fd5-vk4sj\" (UID: \"a87828aa-8038-4643-834a-bbdb78432874\") " pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:48:40.888385 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:40.888357    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jstwt\" (UniqueName: \"kubernetes.io/projected/a87828aa-8038-4643-834a-bbdb78432874-kube-api-access-jstwt\") pod \"rate-test-model-35042a-kserve-7f59b44fd5-vk4sj\" (UID: \"a87828aa-8038-4643-834a-bbdb78432874\") " pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:48:40.888528 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:40.888450    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-model-cache\") pod \"rate-test-model-35042a-kserve-7f59b44fd5-vk4sj\" (UID: \"a87828aa-8038-4643-834a-bbdb78432874\") " pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:48:40.888528 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:40.888494    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-tmp-dir\") pod \"rate-test-model-35042a-kserve-7f59b44fd5-vk4sj\" (UID: \"a87828aa-8038-4643-834a-bbdb78432874\") " pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:48:40.989512 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:40.989478    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-dshm\") pod \"rate-test-model-35042a-kserve-7f59b44fd5-vk4sj\" (UID: \"a87828aa-8038-4643-834a-bbdb78432874\") " pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:48:40.989710 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:40.989539    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jstwt\" (UniqueName: \"kubernetes.io/projected/a87828aa-8038-4643-834a-bbdb78432874-kube-api-access-jstwt\") pod \"rate-test-model-35042a-kserve-7f59b44fd5-vk4sj\" (UID: \"a87828aa-8038-4643-834a-bbdb78432874\") " pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:48:40.989710 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:40.989565    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-model-cache\") pod \"rate-test-model-35042a-kserve-7f59b44fd5-vk4sj\" (UID: \"a87828aa-8038-4643-834a-bbdb78432874\") " pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:48:40.989710 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:40.989585    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-tmp-dir\") pod \"rate-test-model-35042a-kserve-7f59b44fd5-vk4sj\" (UID: \"a87828aa-8038-4643-834a-bbdb78432874\") " pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:48:40.989710 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:40.989611    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-home\") pod \"rate-test-model-35042a-kserve-7f59b44fd5-vk4sj\" (UID: \"a87828aa-8038-4643-834a-bbdb78432874\") " pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:48:40.989710 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:40.989652    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a87828aa-8038-4643-834a-bbdb78432874-tls-certs\") pod \"rate-test-model-35042a-kserve-7f59b44fd5-vk4sj\" (UID: \"a87828aa-8038-4643-834a-bbdb78432874\") " pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:48:40.990007 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:40.989961    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-tmp-dir\") pod \"rate-test-model-35042a-kserve-7f59b44fd5-vk4sj\" (UID: \"a87828aa-8038-4643-834a-bbdb78432874\") " pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:48:40.990086 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:40.990016    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-model-cache\") pod \"rate-test-model-35042a-kserve-7f59b44fd5-vk4sj\" (UID: \"a87828aa-8038-4643-834a-bbdb78432874\") " pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:48:40.990452 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:40.990428    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-home\") pod \"rate-test-model-35042a-kserve-7f59b44fd5-vk4sj\" (UID: \"a87828aa-8038-4643-834a-bbdb78432874\") " pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:48:40.991812 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:40.991790    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-dshm\") pod \"rate-test-model-35042a-kserve-7f59b44fd5-vk4sj\" (UID: \"a87828aa-8038-4643-834a-bbdb78432874\") " pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:48:40.992365 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:40.992344    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a87828aa-8038-4643-834a-bbdb78432874-tls-certs\") pod \"rate-test-model-35042a-kserve-7f59b44fd5-vk4sj\" (UID: \"a87828aa-8038-4643-834a-bbdb78432874\") " pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:48:41.017575 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:41.011784    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jstwt\" (UniqueName: \"kubernetes.io/projected/a87828aa-8038-4643-834a-bbdb78432874-kube-api-access-jstwt\") pod \"rate-test-model-35042a-kserve-7f59b44fd5-vk4sj\" (UID: \"a87828aa-8038-4643-834a-bbdb78432874\") " pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:48:41.099774 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:41.099675    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:48:41.430001 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:41.429969    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"]
Oct 30 01:48:41.431136 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:48:41.431110    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda87828aa_8038_4643_834a_bbdb78432874.slice/crio-9e9c578f6bdf58e5916773a66f558e041ac9891f0d8a31405853d0402a3bc3ee WatchSource:0}: Error finding container 9e9c578f6bdf58e5916773a66f558e041ac9891f0d8a31405853d0402a3bc3ee: Status 404 returned error can't find the container with id 9e9c578f6bdf58e5916773a66f558e041ac9891f0d8a31405853d0402a3bc3ee
Oct 30 01:48:42.297355 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:42.297317    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj" event={"ID":"a87828aa-8038-4643-834a-bbdb78432874","Type":"ContainerStarted","Data":"d1cfd16463abd2603a3cbb6ba56e277eeea77e3af36678edae7cf1344fdaf390"}
Oct 30 01:48:42.297769 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:42.297368    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj" event={"ID":"a87828aa-8038-4643-834a-bbdb78432874","Type":"ContainerStarted","Data":"9e9c578f6bdf58e5916773a66f558e041ac9891f0d8a31405853d0402a3bc3ee"}
Oct 30 01:48:42.338698 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:42.338645    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj" podStartSLOduration=2.338631026 podStartE2EDuration="2.338631026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:42.335470662 +0000 UTC m=+4034.673572597" watchObservedRunningTime="2026-10-30 01:48:42.338631026 +0000 UTC m=+4034.676732936"
Oct 30 01:48:43.100156 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:43.100123    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:48:49.040885 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:48:49.040842    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:48:51.100831 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:51.100795    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:48:51.114177 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:51.114148    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:48:51.420852 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:48:51.420768    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:49:19.043578 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:49:19.043540    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:49:49.044857 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:49:49.044775    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:50:19.041441 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:50:19.041401    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:50:34.868610 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:34.868568    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"]
Oct 30 01:50:34.869013 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:34.868800    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2" podUID="f683790c-2caa-4e0f-8b1a-f72ebd1ab40f" containerName="istio-proxy" containerID="cri-o://f2d3cc045e2d122d599fd0579dc14e66031bfb169227283ebc6d0e1536dadea1" gracePeriod=30
Oct 30 01:50:40.126490 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.126462    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:50:40.143311 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.143287    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-kube-api-access-vn48b\" (UniqueName: \"kubernetes.io/projected/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-kube-api-access-vn48b\") pod \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") "
Oct 30 01:50:40.143427 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.143336    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istiod-ca-cert\") pod \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") "
Oct 30 01:50:40.143489 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.143473    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-workload-socket\") pod \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") "
Oct 30 01:50:40.143552 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.143537    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-podinfo\") pod \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") "
Oct 30 01:50:40.143616 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.143586    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-data\") pod \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") "
Oct 30 01:50:40.143731 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.143617    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-workload-certs\") pod \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") "
Oct 30 01:50:40.143731 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.143668    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-envoy\") pod \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") "
Oct 30 01:50:40.143731 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.143694    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-credential-socket\") pod \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") "
Oct 30 01:50:40.143903 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.143761    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-workload-socket" pod "f683790c-2caa-4e0f-8b1a-f72ebd1ab40f" (UID: "f683790c-2caa-4e0f-8b1a-f72ebd1ab40f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:50:40.143903 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.143843    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-token\" (UniqueName: \"kubernetes.io/projected/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-token\") pod \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\" (UID: \"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f\") "
Oct 30 01:50:40.144012 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.143873    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-data" pod "f683790c-2caa-4e0f-8b1a-f72ebd1ab40f" (UID: "f683790c-2caa-4e0f-8b1a-f72ebd1ab40f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:50:40.144012 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.143937    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-workload-certs" pod "f683790c-2caa-4e0f-8b1a-f72ebd1ab40f" (UID: "f683790c-2caa-4e0f-8b1a-f72ebd1ab40f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:50:40.144012 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.143988    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istiod-ca-cert" pod "f683790c-2caa-4e0f-8b1a-f72ebd1ab40f" (UID: "f683790c-2caa-4e0f-8b1a-f72ebd1ab40f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:50:40.144235 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.144099    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-credential-socket" pod "f683790c-2caa-4e0f-8b1a-f72ebd1ab40f" (UID: "f683790c-2caa-4e0f-8b1a-f72ebd1ab40f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:50:40.144523 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.144493    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:50:40.144640 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.144529    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:50:40.144640 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.144544    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:50:40.144640 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.144556    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:50:40.144640 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.144571    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:50:40.145742 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.145709    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-kube-api-access-vn48b" pod "f683790c-2caa-4e0f-8b1a-f72ebd1ab40f" (UID: "f683790c-2caa-4e0f-8b1a-f72ebd1ab40f"). InnerVolumeSpecName "kube-api-access-vn48b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:50:40.145830 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.145814    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-podinfo" pod "f683790c-2caa-4e0f-8b1a-f72ebd1ab40f" (UID: "f683790c-2caa-4e0f-8b1a-f72ebd1ab40f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:50:40.145884 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.145832    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-envoy" pod "f683790c-2caa-4e0f-8b1a-f72ebd1ab40f" (UID: "f683790c-2caa-4e0f-8b1a-f72ebd1ab40f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:50:40.146024 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.146007    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-token" pod "f683790c-2caa-4e0f-8b1a-f72ebd1ab40f" (UID: "f683790c-2caa-4e0f-8b1a-f72ebd1ab40f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:50:40.195160 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.195039    4864 generic.go:356] "Generic (PLEG): container finished" podID="f683790c-2caa-4e0f-8b1a-f72ebd1ab40f" containerID="f2d3cc045e2d122d599fd0579dc14e66031bfb169227283ebc6d0e1536dadea1" exitCode=0
Oct 30 01:50:40.195443 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.195133    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"
Oct 30 01:50:40.195583 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.195131    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2" event={"ID":"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f","Type":"ContainerDied","Data":"f2d3cc045e2d122d599fd0579dc14e66031bfb169227283ebc6d0e1536dadea1"}
Oct 30 01:50:40.195678 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.195656    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2" event={"ID":"f683790c-2caa-4e0f-8b1a-f72ebd1ab40f","Type":"ContainerDied","Data":"2d601e6c1071c41bb8207b393d1e7c4c01de11fb1bbd548a516d5edf7a8232fd"}
Oct 30 01:50:40.195744 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.195700    4864 scope.go:122] "RemoveContainer" containerID="f2d3cc045e2d122d599fd0579dc14e66031bfb169227283ebc6d0e1536dadea1"
Oct 30 01:50:40.210388 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.210360    4864 scope.go:122] "RemoveContainer" containerID="797f988c5df9994ff609d35ce471fca2e4dbbbc736c08ca1e5f8d1fa7d89bd7c"
Oct 30 01:50:40.220228 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.220179    4864 scope.go:122] "RemoveContainer" containerID="f2d3cc045e2d122d599fd0579dc14e66031bfb169227283ebc6d0e1536dadea1"
Oct 30 01:50:40.220519 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:50:40.220496    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2d3cc045e2d122d599fd0579dc14e66031bfb169227283ebc6d0e1536dadea1\": container with ID starting with f2d3cc045e2d122d599fd0579dc14e66031bfb169227283ebc6d0e1536dadea1 not found: ID does not exist" containerID="f2d3cc045e2d122d599fd0579dc14e66031bfb169227283ebc6d0e1536dadea1"
Oct 30 01:50:40.220641 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.220525    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2d3cc045e2d122d599fd0579dc14e66031bfb169227283ebc6d0e1536dadea1"} err="failed to get container status \"f2d3cc045e2d122d599fd0579dc14e66031bfb169227283ebc6d0e1536dadea1\": rpc error: code = NotFound desc = could not find container \"f2d3cc045e2d122d599fd0579dc14e66031bfb169227283ebc6d0e1536dadea1\": container with ID starting with f2d3cc045e2d122d599fd0579dc14e66031bfb169227283ebc6d0e1536dadea1 not found: ID does not exist"
Oct 30 01:50:40.220641 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.220543    4864 scope.go:122] "RemoveContainer" containerID="797f988c5df9994ff609d35ce471fca2e4dbbbc736c08ca1e5f8d1fa7d89bd7c"
Oct 30 01:50:40.221143 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:50:40.221057    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"797f988c5df9994ff609d35ce471fca2e4dbbbc736c08ca1e5f8d1fa7d89bd7c\": container with ID starting with 797f988c5df9994ff609d35ce471fca2e4dbbbc736c08ca1e5f8d1fa7d89bd7c not found: ID does not exist" containerID="797f988c5df9994ff609d35ce471fca2e4dbbbc736c08ca1e5f8d1fa7d89bd7c"
Oct 30 01:50:40.221251 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.221150    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797f988c5df9994ff609d35ce471fca2e4dbbbc736c08ca1e5f8d1fa7d89bd7c"} err="failed to get container status \"797f988c5df9994ff609d35ce471fca2e4dbbbc736c08ca1e5f8d1fa7d89bd7c\": rpc error: code = NotFound desc = could not find container \"797f988c5df9994ff609d35ce471fca2e4dbbbc736c08ca1e5f8d1fa7d89bd7c\": container with ID starting with 797f988c5df9994ff609d35ce471fca2e4dbbbc736c08ca1e5f8d1fa7d89bd7c not found: ID does not exist"
Oct 30 01:50:40.244096 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.244051    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"]
Oct 30 01:50:40.246023 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.245998    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:50:40.246023 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.246021    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:50:40.246179 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.246032    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:50:40.246179 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.246040    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn48b\" (UniqueName: \"kubernetes.io/projected/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f-kube-api-access-vn48b\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:50:40.250935 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:40.250910    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e6fc524e-openshift-default-69ff46fc97-ltxd2"]
Oct 30 01:50:41.801014 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:41.800981    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f683790c-2caa-4e0f-8b1a-f72ebd1ab40f" path="/var/lib/kubelet/pods/f683790c-2caa-4e0f-8b1a-f72ebd1ab40f/volumes"
Oct 30 01:50:47.080852 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:47.080818    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl"]
Oct 30 01:50:47.081407 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:47.081036    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl" podUID="44072e00-e386-4e3e-9d8a-3087d4dc9c16" containerName="maas-api" containerID="cri-o://047d33e798e5fd7ab02068475a674018cb184b317353f10780fc3403ab67901b" gracePeriod=30
Oct 30 01:50:47.111254 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:47.111218    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl" podUID="44072e00-e386-4e3e-9d8a-3087d4dc9c16" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.136:8443/health\": dial tcp 10.132.0.136:8443: connect: connection refused"
Oct 30 01:50:47.293996 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:47.293664    4864 generic.go:356] "Generic (PLEG): container finished" podID="44072e00-e386-4e3e-9d8a-3087d4dc9c16" containerID="047d33e798e5fd7ab02068475a674018cb184b317353f10780fc3403ab67901b" exitCode=0
Oct 30 01:50:47.293996 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:47.293919    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl" event={"ID":"44072e00-e386-4e3e-9d8a-3087d4dc9c16","Type":"ContainerDied","Data":"047d33e798e5fd7ab02068475a674018cb184b317353f10780fc3403ab67901b"}
Oct 30 01:50:47.397156 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:47.397130    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl"
Oct 30 01:50:47.464828 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:47.464784    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44072e00-e386-4e3e-9d8a-3087d4dc9c16-kube-api-access-28x9q\" (UniqueName: \"kubernetes.io/projected/44072e00-e386-4e3e-9d8a-3087d4dc9c16-kube-api-access-28x9q\") pod \"44072e00-e386-4e3e-9d8a-3087d4dc9c16\" (UID: \"44072e00-e386-4e3e-9d8a-3087d4dc9c16\") "
Oct 30 01:50:47.465008 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:47.464839    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44072e00-e386-4e3e-9d8a-3087d4dc9c16-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44072e00-e386-4e3e-9d8a-3087d4dc9c16-openshift-service-ca\") pod \"44072e00-e386-4e3e-9d8a-3087d4dc9c16\" (UID: \"44072e00-e386-4e3e-9d8a-3087d4dc9c16\") "
Oct 30 01:50:47.465008 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:47.464885    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44072e00-e386-4e3e-9d8a-3087d4dc9c16-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44072e00-e386-4e3e-9d8a-3087d4dc9c16-maas-api-tls\") pod \"44072e00-e386-4e3e-9d8a-3087d4dc9c16\" (UID: \"44072e00-e386-4e3e-9d8a-3087d4dc9c16\") "
Oct 30 01:50:47.465369 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:47.465140    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44072e00-e386-4e3e-9d8a-3087d4dc9c16-metrics-certs\" (UniqueName: \"kubernetes.io/secret/44072e00-e386-4e3e-9d8a-3087d4dc9c16-metrics-certs\") pod \"44072e00-e386-4e3e-9d8a-3087d4dc9c16\" (UID: \"44072e00-e386-4e3e-9d8a-3087d4dc9c16\") "
Oct 30 01:50:47.466391 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:47.466357    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44072e00-e386-4e3e-9d8a-3087d4dc9c16-openshift-service-ca" pod "44072e00-e386-4e3e-9d8a-3087d4dc9c16" (UID: "44072e00-e386-4e3e-9d8a-3087d4dc9c16"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:50:47.473619 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:47.468425    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44072e00-e386-4e3e-9d8a-3087d4dc9c16-metrics-certs" pod "44072e00-e386-4e3e-9d8a-3087d4dc9c16" (UID: "44072e00-e386-4e3e-9d8a-3087d4dc9c16"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:50:47.492348 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:47.490458    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44072e00-e386-4e3e-9d8a-3087d4dc9c16-kube-api-access-28x9q" pod "44072e00-e386-4e3e-9d8a-3087d4dc9c16" (UID: "44072e00-e386-4e3e-9d8a-3087d4dc9c16"). InnerVolumeSpecName "kube-api-access-28x9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:50:47.492348 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:47.490953    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44072e00-e386-4e3e-9d8a-3087d4dc9c16-maas-api-tls" pod "44072e00-e386-4e3e-9d8a-3087d4dc9c16" (UID: "44072e00-e386-4e3e-9d8a-3087d4dc9c16"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:50:47.567525 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:47.567491    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28x9q\" (UniqueName: \"kubernetes.io/projected/44072e00-e386-4e3e-9d8a-3087d4dc9c16-kube-api-access-28x9q\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:50:47.567525 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:47.567519    4864 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44072e00-e386-4e3e-9d8a-3087d4dc9c16-openshift-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:50:47.567525 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:47.567529    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44072e00-e386-4e3e-9d8a-3087d4dc9c16-maas-api-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:50:47.567749 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:47.567538    4864 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44072e00-e386-4e3e-9d8a-3087d4dc9c16-metrics-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:50:48.303004 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:48.302976    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl"
Oct 30 01:50:48.303004 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:48.302996    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl" event={"ID":"44072e00-e386-4e3e-9d8a-3087d4dc9c16","Type":"ContainerDied","Data":"bf6f4fd9ab974f90c316eee6c45ecbd2475bf8aed047e1672d90098d738a9c46"}
Oct 30 01:50:48.303509 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:48.303052    4864 scope.go:122] "RemoveContainer" containerID="047d33e798e5fd7ab02068475a674018cb184b317353f10780fc3403ab67901b"
Oct 30 01:50:48.352167 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:48.352131    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl"]
Oct 30 01:50:48.367716 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:48.367686    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c7bba9-76f9d4fbb8-8vthl"]
Oct 30 01:50:49.039615 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:50:49.039580    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:50:49.795824 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:50:49.795785    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44072e00-e386-4e3e-9d8a-3087d4dc9c16" path="/var/lib/kubelet/pods/44072e00-e386-4e3e-9d8a-3087d4dc9c16/volumes"
Oct 30 01:51:14.231992 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.231895    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"]
Oct 30 01:51:14.232595 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.232577    4864 cpu_manager.go:414] "containerMap: removing container" podUID="44072e00-e386-4e3e-9d8a-3087d4dc9c16" containerName="maas-api"
Oct 30 01:51:14.232650 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.232596    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44072e00-e386-4e3e-9d8a-3087d4dc9c16" containerName="maas-api"
Oct 30 01:51:14.232650 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.232638    4864 cpu_manager.go:414] "containerMap: removing container" podUID="f683790c-2caa-4e0f-8b1a-f72ebd1ab40f" containerName="istio-proxy"
Oct 30 01:51:14.232650 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.232644    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f683790c-2caa-4e0f-8b1a-f72ebd1ab40f" containerName="istio-proxy"
Oct 30 01:51:14.232749 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.232661    4864 cpu_manager.go:414] "containerMap: removing container" podUID="f683790c-2caa-4e0f-8b1a-f72ebd1ab40f" containerName="istio-proxy"
Oct 30 01:51:14.232749 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.232667    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f683790c-2caa-4e0f-8b1a-f72ebd1ab40f" containerName="istio-proxy"
Oct 30 01:51:14.232820 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.232773    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="44072e00-e386-4e3e-9d8a-3087d4dc9c16" containerName="maas-api"
Oct 30 01:51:14.232820 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.232787    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="f683790c-2caa-4e0f-8b1a-f72ebd1ab40f" containerName="istio-proxy"
Oct 30 01:51:14.232820 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.232799    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="f683790c-2caa-4e0f-8b1a-f72ebd1ab40f" containerName="istio-proxy"
Oct 30 01:51:14.236448 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.236426    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:51:14.261099 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.260968    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-bc127f\"/\"kube-root-ca.crt\""
Oct 30 01:51:14.261733 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.261710    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-bc127f\"/\"openshift-service-ca.crt\""
Oct 30 01:51:14.261869 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.261847    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-bc127f\"/\"rate-test-model-bc127f-kserve-self-signed-certs\""
Oct 30 01:51:14.261998 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.261901    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-bc127f\"/\"default-dockercfg-qjx7n\""
Oct 30 01:51:14.265855 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.265822    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"]
Oct 30 01:51:14.364272 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.364233    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ckwt\" (UniqueName: \"kubernetes.io/projected/e8667b92-fdd3-49f8-bd1b-541d378530ef-kube-api-access-7ckwt\") pod \"rate-test-model-bc127f-kserve-5fd695c845-g8ppz\" (UID: \"e8667b92-fdd3-49f8-bd1b-541d378530ef\") " pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:51:14.364450 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.364400    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-tmp-dir\") pod \"rate-test-model-bc127f-kserve-5fd695c845-g8ppz\" (UID: \"e8667b92-fdd3-49f8-bd1b-541d378530ef\") " pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:51:14.364527 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.364512    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8667b92-fdd3-49f8-bd1b-541d378530ef-tls-certs\") pod \"rate-test-model-bc127f-kserve-5fd695c845-g8ppz\" (UID: \"e8667b92-fdd3-49f8-bd1b-541d378530ef\") " pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:51:14.364579 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.364538    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-model-cache\") pod \"rate-test-model-bc127f-kserve-5fd695c845-g8ppz\" (UID: \"e8667b92-fdd3-49f8-bd1b-541d378530ef\") " pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:51:14.364637 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.364620    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-dshm\") pod \"rate-test-model-bc127f-kserve-5fd695c845-g8ppz\" (UID: \"e8667b92-fdd3-49f8-bd1b-541d378530ef\") " pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:51:14.364677 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.364658    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-home\") pod \"rate-test-model-bc127f-kserve-5fd695c845-g8ppz\" (UID: \"e8667b92-fdd3-49f8-bd1b-541d378530ef\") " pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:51:14.466294 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.466253    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-dshm\") pod \"rate-test-model-bc127f-kserve-5fd695c845-g8ppz\" (UID: \"e8667b92-fdd3-49f8-bd1b-541d378530ef\") " pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:51:14.466489 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.466315    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-home\") pod \"rate-test-model-bc127f-kserve-5fd695c845-g8ppz\" (UID: \"e8667b92-fdd3-49f8-bd1b-541d378530ef\") " pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:51:14.466489 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.466355    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ckwt\" (UniqueName: \"kubernetes.io/projected/e8667b92-fdd3-49f8-bd1b-541d378530ef-kube-api-access-7ckwt\") pod \"rate-test-model-bc127f-kserve-5fd695c845-g8ppz\" (UID: \"e8667b92-fdd3-49f8-bd1b-541d378530ef\") " pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:51:14.466489 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.466457    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-tmp-dir\") pod \"rate-test-model-bc127f-kserve-5fd695c845-g8ppz\" (UID: \"e8667b92-fdd3-49f8-bd1b-541d378530ef\") " pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:51:14.466676 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.466530    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8667b92-fdd3-49f8-bd1b-541d378530ef-tls-certs\") pod \"rate-test-model-bc127f-kserve-5fd695c845-g8ppz\" (UID: \"e8667b92-fdd3-49f8-bd1b-541d378530ef\") " pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:51:14.466676 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.466552    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-model-cache\") pod \"rate-test-model-bc127f-kserve-5fd695c845-g8ppz\" (UID: \"e8667b92-fdd3-49f8-bd1b-541d378530ef\") " pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:51:14.466823 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.466800    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-home\") pod \"rate-test-model-bc127f-kserve-5fd695c845-g8ppz\" (UID: \"e8667b92-fdd3-49f8-bd1b-541d378530ef\") " pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:51:14.466864 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.466852    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-tmp-dir\") pod \"rate-test-model-bc127f-kserve-5fd695c845-g8ppz\" (UID: \"e8667b92-fdd3-49f8-bd1b-541d378530ef\") " pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:51:14.466994 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.466971    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-model-cache\") pod \"rate-test-model-bc127f-kserve-5fd695c845-g8ppz\" (UID: \"e8667b92-fdd3-49f8-bd1b-541d378530ef\") " pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:51:14.468608 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.468584    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-dshm\") pod \"rate-test-model-bc127f-kserve-5fd695c845-g8ppz\" (UID: \"e8667b92-fdd3-49f8-bd1b-541d378530ef\") " pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:51:14.468918 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.468899    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8667b92-fdd3-49f8-bd1b-541d378530ef-tls-certs\") pod \"rate-test-model-bc127f-kserve-5fd695c845-g8ppz\" (UID: \"e8667b92-fdd3-49f8-bd1b-541d378530ef\") " pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:51:14.555944 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.553258    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ckwt\" (UniqueName: \"kubernetes.io/projected/e8667b92-fdd3-49f8-bd1b-541d378530ef-kube-api-access-7ckwt\") pod \"rate-test-model-bc127f-kserve-5fd695c845-g8ppz\" (UID: \"e8667b92-fdd3-49f8-bd1b-541d378530ef\") " pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:51:14.555944 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.555395    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:51:14.879123 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:14.879094    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"]
Oct 30 01:51:14.881447 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:51:14.881421    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8667b92_fdd3_49f8_bd1b_541d378530ef.slice/crio-8585aa526ecefb1baea866d03a35a14be32381c913f8f736fa87b13feded584e WatchSource:0}: Error finding container 8585aa526ecefb1baea866d03a35a14be32381c913f8f736fa87b13feded584e: Status 404 returned error can't find the container with id 8585aa526ecefb1baea866d03a35a14be32381c913f8f736fa87b13feded584e
Oct 30 01:51:15.565496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:15.565460    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz" event={"ID":"e8667b92-fdd3-49f8-bd1b-541d378530ef","Type":"ContainerStarted","Data":"30d8cef50595ded612d2e1625e9428f09800de4c7993e29666d4f18aa3473529"}
Oct 30 01:51:15.565999 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:15.565508    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz" event={"ID":"e8667b92-fdd3-49f8-bd1b-541d378530ef","Type":"ContainerStarted","Data":"8585aa526ecefb1baea866d03a35a14be32381c913f8f736fa87b13feded584e"}
Oct 30 01:51:15.607404 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:15.607350    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz" podStartSLOduration=1.607336275 podStartE2EDuration="1.607336275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:15.606345013 +0000 UTC m=+4187.944446922" watchObservedRunningTime="2026-10-30 01:51:15.607336275 +0000 UTC m=+4187.945438185"
Oct 30 01:51:16.555800 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:16.555762    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:51:19.040147 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:51:19.040098    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:51:24.555891 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:24.555855    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:51:24.571842 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:24.571816    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:51:24.636017 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:24.635987    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:51:28.659366 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:28.659338    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:51:28.668959 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:28.668934    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:51:43.707100 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:43.707055    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"]
Oct 30 01:51:43.707579 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:43.707330    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4" podUID="3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" containerName="istio-proxy" containerID="cri-o://2cf55743e6bde2042e678d38fbe8ebca98fb67325bfe10affa97104f39d419c8" gracePeriod=30
Oct 30 01:51:45.401158 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:45.401118    4864 patch_prober.go:29] interesting pod/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:51:45.401605 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:45.401193    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4" podUID="3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:51:46.952585 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:46.952549    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"]
Oct 30 01:51:46.953005 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:46.952729    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9" podUID="63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f" containerName="discovery" containerID="cri-o://4ea8d039f683a3598bb38cf1c942b68a4872dfb5db511d6c9490343d07c5f2fd" gracePeriod=30
Oct 30 01:51:47.236777 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.236751    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 01:51:47.306358 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.306322    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-kubeconfig\") pod \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") "
Oct 30 01:51:47.306547 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.306393    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-token\" (UniqueName: \"kubernetes.io/projected/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-token\") pod \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") "
Oct 30 01:51:47.309282 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.306513    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-local-certs\") pod \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") "
Oct 30 01:51:47.309282 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.306963    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-csr-ca-configmap\") pod \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") "
Oct 30 01:51:47.309282 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.307175    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-cacerts\" (UniqueName: \"kubernetes.io/secret/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-cacerts\") pod \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") "
Oct 30 01:51:47.309282 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.307226    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-csr-dns-cert\") pod \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") "
Oct 30 01:51:47.309282 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.307288    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-kube-api-access-wgrq5\" (UniqueName: \"kubernetes.io/projected/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-kube-api-access-wgrq5\") pod \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\" (UID: \"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f\") "
Oct 30 01:51:47.309282 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.309051    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-kubeconfig" pod "63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f" (UID: "63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:51:47.309282 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.309185    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-csr-ca-configmap" pod "63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f" (UID: "63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:51:47.310315 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.310263    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-local-certs" pod "63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f" (UID: "63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:51:47.311554 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.311519    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-token" pod "63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f" (UID: "63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:51:47.312690 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.312654    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-cacerts" pod "63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f" (UID: "63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:51:47.312846 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.312805    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-csr-dns-cert" pod "63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f" (UID: "63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:51:47.313421 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.313387    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-kube-api-access-wgrq5" pod "63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f" (UID: "63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f"). InnerVolumeSpecName "kube-api-access-wgrq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:51:47.408844 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.408806    4864 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-cacerts\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:51:47.408844 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.408842    4864 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-csr-dns-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:51:47.409130 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.408854    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgrq5\" (UniqueName: \"kubernetes.io/projected/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-kube-api-access-wgrq5\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:51:47.409130 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.408864    4864 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-kubeconfig\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:51:47.409130 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.408872    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:51:47.409130 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.408882    4864 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-local-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:51:47.409130 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.408890    4864 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f-istio-csr-ca-configmap\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:51:47.794790 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.794749    4864 generic.go:356] "Generic (PLEG): container finished" podID="63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f" containerID="4ea8d039f683a3598bb38cf1c942b68a4872dfb5db511d6c9490343d07c5f2fd" exitCode=0
Oct 30 01:51:47.795006 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.794857    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"
Oct 30 01:51:47.797287 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.797261    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9" event={"ID":"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f","Type":"ContainerDied","Data":"4ea8d039f683a3598bb38cf1c942b68a4872dfb5db511d6c9490343d07c5f2fd"}
Oct 30 01:51:47.797409 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.797302    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9" event={"ID":"63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f","Type":"ContainerDied","Data":"e8d731782a217c9576a1c090cbbb6ef18a36eb8ce28f41dff39d04eb34dba8e4"}
Oct 30 01:51:47.797409 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.797348    4864 scope.go:122] "RemoveContainer" containerID="4ea8d039f683a3598bb38cf1c942b68a4872dfb5db511d6c9490343d07c5f2fd"
Oct 30 01:51:47.816279 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.816260    4864 scope.go:122] "RemoveContainer" containerID="4ea8d039f683a3598bb38cf1c942b68a4872dfb5db511d6c9490343d07c5f2fd"
Oct 30 01:51:47.816564 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:51:47.816542    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ea8d039f683a3598bb38cf1c942b68a4872dfb5db511d6c9490343d07c5f2fd\": container with ID starting with 4ea8d039f683a3598bb38cf1c942b68a4872dfb5db511d6c9490343d07c5f2fd not found: ID does not exist" containerID="4ea8d039f683a3598bb38cf1c942b68a4872dfb5db511d6c9490343d07c5f2fd"
Oct 30 01:51:47.816615 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.816572    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea8d039f683a3598bb38cf1c942b68a4872dfb5db511d6c9490343d07c5f2fd"} err="failed to get container status \"4ea8d039f683a3598bb38cf1c942b68a4872dfb5db511d6c9490343d07c5f2fd\": rpc error: code = NotFound desc = could not find container \"4ea8d039f683a3598bb38cf1c942b68a4872dfb5db511d6c9490343d07c5f2fd\": container with ID starting with 4ea8d039f683a3598bb38cf1c942b68a4872dfb5db511d6c9490343d07c5f2fd not found: ID does not exist"
Oct 30 01:51:47.845157 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.845128    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"]
Oct 30 01:51:47.855197 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:47.855170    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6xv9"]
Oct 30 01:51:49.001534 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.001508    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:51:49.039196 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:51:49.039159    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:51:49.134320 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.134286    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-workload-socket\") pod \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") "
Oct 30 01:51:49.134499 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.134376    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-kube-api-access-h8w9t\" (UniqueName: \"kubernetes.io/projected/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-kube-api-access-h8w9t\") pod \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") "
Oct 30 01:51:49.134499 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.134419    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istiod-ca-cert\") pod \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") "
Oct 30 01:51:49.134499 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.134493    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-credential-socket\") pod \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") "
Oct 30 01:51:49.134653 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.134556    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-workload-socket" pod "3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" (UID: "3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:51:49.134653 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.134590    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-podinfo\") pod \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") "
Oct 30 01:51:49.134653 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.134644    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-data\") pod \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") "
Oct 30 01:51:49.134826 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.134675    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-envoy\") pod \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") "
Oct 30 01:51:49.134826 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.134704    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-token\" (UniqueName: \"kubernetes.io/projected/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-token\") pod \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") "
Oct 30 01:51:49.134826 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.134749    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-credential-socket" pod "3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" (UID: "3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:51:49.134826 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.134762    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-workload-certs\") pod \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\" (UID: \"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c\") "
Oct 30 01:51:49.134826 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.134785    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istiod-ca-cert" pod "3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" (UID: "3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:51:49.135068 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.134942    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-data" pod "3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" (UID: "3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:51:49.135200 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.135176    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-workload-certs" pod "3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" (UID: "3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:51:49.135486 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.135458    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:51:49.135586 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.135490    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:51:49.135586 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.135506    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:51:49.135586 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.135521    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:51:49.135586 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.135536    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:51:49.136835 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.136807    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-kube-api-access-h8w9t" pod "3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" (UID: "3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c"). InnerVolumeSpecName "kube-api-access-h8w9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:51:49.136940 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.136835    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-envoy" pod "3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" (UID: "3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:51:49.136940 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.136853    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-token" pod "3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" (UID: "3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:51:49.136940 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.136902    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-podinfo" pod "3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" (UID: "3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:51:49.236867 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.236782    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8w9t\" (UniqueName: \"kubernetes.io/projected/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-kube-api-access-h8w9t\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:51:49.236867 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.236816    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:51:49.236867 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.236827    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:51:49.236867 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.236835    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:51:49.794316 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.794283    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f" path="/var/lib/kubelet/pods/63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f/volumes"
Oct 30 01:51:49.811819 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.811790    4864 generic.go:356] "Generic (PLEG): container finished" podID="3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" containerID="2cf55743e6bde2042e678d38fbe8ebca98fb67325bfe10affa97104f39d419c8" exitCode=0
Oct 30 01:51:49.811947 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.811845    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"
Oct 30 01:51:49.811947 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.811867    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4" event={"ID":"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c","Type":"ContainerDied","Data":"2cf55743e6bde2042e678d38fbe8ebca98fb67325bfe10affa97104f39d419c8"}
Oct 30 01:51:49.811947 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.811912    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4" event={"ID":"3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c","Type":"ContainerDied","Data":"a3764851c176c9358216103842aae91fe5a397c85203948fe90eafcc86a9167b"}
Oct 30 01:51:49.811947 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.811935    4864 scope.go:122] "RemoveContainer" containerID="2cf55743e6bde2042e678d38fbe8ebca98fb67325bfe10affa97104f39d419c8"
Oct 30 01:51:49.822897 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.822876    4864 scope.go:122] "RemoveContainer" containerID="34f56cf60c8df35eae8740bda92891c5f88f45be1bb06a55c2b0ee36f44aa07f"
Oct 30 01:51:49.832392 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.832214    4864 scope.go:122] "RemoveContainer" containerID="2cf55743e6bde2042e678d38fbe8ebca98fb67325bfe10affa97104f39d419c8"
Oct 30 01:51:49.832547 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:51:49.832530    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cf55743e6bde2042e678d38fbe8ebca98fb67325bfe10affa97104f39d419c8\": container with ID starting with 2cf55743e6bde2042e678d38fbe8ebca98fb67325bfe10affa97104f39d419c8 not found: ID does not exist" containerID="2cf55743e6bde2042e678d38fbe8ebca98fb67325bfe10affa97104f39d419c8"
Oct 30 01:51:49.832593 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.832557    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cf55743e6bde2042e678d38fbe8ebca98fb67325bfe10affa97104f39d419c8"} err="failed to get container status \"2cf55743e6bde2042e678d38fbe8ebca98fb67325bfe10affa97104f39d419c8\": rpc error: code = NotFound desc = could not find container \"2cf55743e6bde2042e678d38fbe8ebca98fb67325bfe10affa97104f39d419c8\": container with ID starting with 2cf55743e6bde2042e678d38fbe8ebca98fb67325bfe10affa97104f39d419c8 not found: ID does not exist"
Oct 30 01:51:49.832593 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.832575    4864 scope.go:122] "RemoveContainer" containerID="34f56cf60c8df35eae8740bda92891c5f88f45be1bb06a55c2b0ee36f44aa07f"
Oct 30 01:51:49.832832 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:51:49.832814    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34f56cf60c8df35eae8740bda92891c5f88f45be1bb06a55c2b0ee36f44aa07f\": container with ID starting with 34f56cf60c8df35eae8740bda92891c5f88f45be1bb06a55c2b0ee36f44aa07f not found: ID does not exist" containerID="34f56cf60c8df35eae8740bda92891c5f88f45be1bb06a55c2b0ee36f44aa07f"
Oct 30 01:51:49.832885 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.832837    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34f56cf60c8df35eae8740bda92891c5f88f45be1bb06a55c2b0ee36f44aa07f"} err="failed to get container status \"34f56cf60c8df35eae8740bda92891c5f88f45be1bb06a55c2b0ee36f44aa07f\": rpc error: code = NotFound desc = could not find container \"34f56cf60c8df35eae8740bda92891c5f88f45be1bb06a55c2b0ee36f44aa07f\": container with ID starting with 34f56cf60c8df35eae8740bda92891c5f88f45be1bb06a55c2b0ee36f44aa07f not found: ID does not exist"
Oct 30 01:51:49.870163 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.870130    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"]
Oct 30 01:51:49.880923 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:49.880893    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7bba9-openshift-default-588bb65b97-5dqd4"]
Oct 30 01:51:51.795264 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:51:51.795226    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" path="/var/lib/kubelet/pods/3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c/volumes"
Oct 30 01:52:08.480197 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:08.480159    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7285-4349cd"]
Oct 30 01:52:08.480756 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:08.480742    4864 cpu_manager.go:414] "containerMap: removing container" podUID="3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" containerName="istio-proxy"
Oct 30 01:52:08.480801 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:08.480757    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" containerName="istio-proxy"
Oct 30 01:52:08.480801 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:08.480779    4864 cpu_manager.go:414] "containerMap: removing container" podUID="63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f" containerName="discovery"
Oct 30 01:52:08.480801 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:08.480784    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f" containerName="discovery"
Oct 30 01:52:08.480900 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:08.480802    4864 cpu_manager.go:414] "containerMap: removing container" podUID="3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" containerName="istio-proxy"
Oct 30 01:52:08.480900 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:08.480807    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" containerName="istio-proxy"
Oct 30 01:52:08.480966 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:08.480900    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" containerName="istio-proxy"
Oct 30 01:52:08.480966 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:08.480915    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf8fc6a-2b8a-420e-83e1-b5ba04312b9c" containerName="istio-proxy"
Oct 30 01:52:08.480966 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:08.480925    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d4a7e1-5c2b-41a4-b2e0-73a3048ec79f" containerName="discovery"
Oct 30 01:52:08.484326 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:08.484307    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-4349cd"
Oct 30 01:52:08.501376 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:08.501347    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7285-4349cd"]
Oct 30 01:52:08.506507 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:08.506481    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4z6vl\""
Oct 30 01:52:08.597607 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:08.597573    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6t6k\" (UniqueName: \"kubernetes.io/projected/6492d2c9-d332-4048-9bed-a904417ec3b3-kube-api-access-f6t6k\") pod \"test-curl-7285-4349cd\" (UID: \"6492d2c9-d332-4048-9bed-a904417ec3b3\") " pod="opendatahub/test-curl-7285-4349cd"
Oct 30 01:52:08.698949 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:08.698912    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6t6k\" (UniqueName: \"kubernetes.io/projected/6492d2c9-d332-4048-9bed-a904417ec3b3-kube-api-access-f6t6k\") pod \"test-curl-7285-4349cd\" (UID: \"6492d2c9-d332-4048-9bed-a904417ec3b3\") " pod="opendatahub/test-curl-7285-4349cd"
Oct 30 01:52:08.721635 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:08.721600    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6t6k\" (UniqueName: \"kubernetes.io/projected/6492d2c9-d332-4048-9bed-a904417ec3b3-kube-api-access-f6t6k\") pod \"test-curl-7285-4349cd\" (UID: \"6492d2c9-d332-4048-9bed-a904417ec3b3\") " pod="opendatahub/test-curl-7285-4349cd"
Oct 30 01:52:08.797533 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:08.797450    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-4349cd"
Oct 30 01:52:09.095786 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:09.095758    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7285-4349cd"]
Oct 30 01:52:09.097914 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:52:09.097883    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6492d2c9_d332_4048_9bed_a904417ec3b3.slice/crio-657cd5f6297813189c5eabc35e2906b94695f07a3538e5b9bd795e9e644f7741 WatchSource:0}: Error finding container 657cd5f6297813189c5eabc35e2906b94695f07a3538e5b9bd795e9e644f7741: Status 404 returned error can't find the container with id 657cd5f6297813189c5eabc35e2906b94695f07a3538e5b9bd795e9e644f7741
Oct 30 01:52:09.950945 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:09.950910    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7285-4349cd" event={"ID":"6492d2c9-d332-4048-9bed-a904417ec3b3","Type":"ContainerStarted","Data":"987cf56aee8f884391327f4a8fd25070a566c9506d72c262c51eb4face47eede"}
Oct 30 01:52:09.950945 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:09.950954    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7285-4349cd" event={"ID":"6492d2c9-d332-4048-9bed-a904417ec3b3","Type":"ContainerStarted","Data":"657cd5f6297813189c5eabc35e2906b94695f07a3538e5b9bd795e9e644f7741"}
Oct 30 01:52:09.988118 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:09.988044    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7285-4349cd" podStartSLOduration=1.9880305329999999 podStartE2EDuration="1.988030533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:09.986318853 +0000 UTC m=+4242.324420764" watchObservedRunningTime="2026-10-30 01:52:09.988030533 +0000 UTC m=+4242.326132437"
Oct 30 01:52:10.900333 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:10.900299    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7285-4349cd"]
Oct 30 01:52:10.913353 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:10.913328    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7285-4349cd"]
Oct 30 01:52:11.063251 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:11.063218    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7285-286c6e"]
Oct 30 01:52:11.063908 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:11.063893    4864 cpu_manager.go:414] "containerMap: removing container" podUID="6492d2c9-d332-4048-9bed-a904417ec3b3" containerName="test-curl-7285-4349cd"
Oct 30 01:52:11.063908 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:11.063908    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6492d2c9-d332-4048-9bed-a904417ec3b3" containerName="test-curl-7285-4349cd"
Oct 30 01:52:11.064023 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:11.064014    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="6492d2c9-d332-4048-9bed-a904417ec3b3" containerName="test-curl-7285-4349cd"
Oct 30 01:52:11.067497 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:11.067480    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-286c6e"
Oct 30 01:52:11.086712 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:11.086688    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7285-286c6e"]
Oct 30 01:52:11.228828 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:11.228789    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmspf\" (UniqueName: \"kubernetes.io/projected/3034998b-5403-4d4d-91e4-f9185beb21c5-kube-api-access-cmspf\") pod \"test-curl-7285-286c6e\" (UID: \"3034998b-5403-4d4d-91e4-f9185beb21c5\") " pod="opendatahub/test-curl-7285-286c6e"
Oct 30 01:52:11.330374 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:11.330334    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmspf\" (UniqueName: \"kubernetes.io/projected/3034998b-5403-4d4d-91e4-f9185beb21c5-kube-api-access-cmspf\") pod \"test-curl-7285-286c6e\" (UID: \"3034998b-5403-4d4d-91e4-f9185beb21c5\") " pod="opendatahub/test-curl-7285-286c6e"
Oct 30 01:52:11.353646 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:11.353610    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmspf\" (UniqueName: \"kubernetes.io/projected/3034998b-5403-4d4d-91e4-f9185beb21c5-kube-api-access-cmspf\") pod \"test-curl-7285-286c6e\" (UID: \"3034998b-5403-4d4d-91e4-f9185beb21c5\") " pod="opendatahub/test-curl-7285-286c6e"
Oct 30 01:52:11.379518 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:11.379491    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-286c6e"
Oct 30 01:52:11.685663 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:11.685637    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7285-286c6e"]
Oct 30 01:52:11.687412 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:52:11.687383    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3034998b_5403_4d4d_91e4_f9185beb21c5.slice/crio-35c5bb92980940b78c1ad5ac5b10d92a604a1567e6353cd30c53b261cdbbf355 WatchSource:0}: Error finding container 35c5bb92980940b78c1ad5ac5b10d92a604a1567e6353cd30c53b261cdbbf355: Status 404 returned error can't find the container with id 35c5bb92980940b78c1ad5ac5b10d92a604a1567e6353cd30c53b261cdbbf355
Oct 30 01:52:11.966725 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:11.966644    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7285-286c6e" event={"ID":"3034998b-5403-4d4d-91e4-f9185beb21c5","Type":"ContainerStarted","Data":"56c88e0061f72dbc4f6e69c1eb48da602841cf7673205ac5997dbe35ca549cf3"}
Oct 30 01:52:11.966725 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:11.966734    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7285-286c6e" event={"ID":"3034998b-5403-4d4d-91e4-f9185beb21c5","Type":"ContainerStarted","Data":"35c5bb92980940b78c1ad5ac5b10d92a604a1567e6353cd30c53b261cdbbf355"}
Oct 30 01:52:11.966986 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:11.966857    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7285-4349cd" podUID="6492d2c9-d332-4048-9bed-a904417ec3b3" containerName="test-curl-7285-4349cd" containerID="cri-o://987cf56aee8f884391327f4a8fd25070a566c9506d72c262c51eb4face47eede" gracePeriod=2
Oct 30 01:52:12.004148 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:12.004066    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7285-286c6e" podStartSLOduration=1.004052985 podStartE2EDuration="1.004052985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:12.002209973 +0000 UTC m=+4244.340311866" watchObservedRunningTime="2026-10-30 01:52:12.004052985 +0000 UTC m=+4244.342154895"
Oct 30 01:52:12.915649 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:12.915611    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7285-286c6e"]
Oct 30 01:52:12.920312 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:12.920272    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7285-286c6e"]
Oct 30 01:52:13.981509 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:13.981479    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7285-286c6e" podUID="3034998b-5403-4d4d-91e4-f9185beb21c5" containerName="test-curl-7285-286c6e" containerID="cri-o://56c88e0061f72dbc4f6e69c1eb48da602841cf7673205ac5997dbe35ca549cf3" gracePeriod=2
Oct 30 01:52:14.151008 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:14.150979    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-4349cd"
Oct 30 01:52:14.268983 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:14.268893    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6492d2c9-d332-4048-9bed-a904417ec3b3-kube-api-access-f6t6k\" (UniqueName: \"kubernetes.io/projected/6492d2c9-d332-4048-9bed-a904417ec3b3-kube-api-access-f6t6k\") pod \"6492d2c9-d332-4048-9bed-a904417ec3b3\" (UID: \"6492d2c9-d332-4048-9bed-a904417ec3b3\") "
Oct 30 01:52:14.271253 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:14.271217    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6492d2c9-d332-4048-9bed-a904417ec3b3-kube-api-access-f6t6k" pod "6492d2c9-d332-4048-9bed-a904417ec3b3" (UID: "6492d2c9-d332-4048-9bed-a904417ec3b3"). InnerVolumeSpecName "kube-api-access-f6t6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:14.370617 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:14.370579    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6t6k\" (UniqueName: \"kubernetes.io/projected/6492d2c9-d332-4048-9bed-a904417ec3b3-kube-api-access-f6t6k\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:52:14.990136 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:14.990104    4864 generic.go:356] "Generic (PLEG): container finished" podID="6492d2c9-d332-4048-9bed-a904417ec3b3" containerID="987cf56aee8f884391327f4a8fd25070a566c9506d72c262c51eb4face47eede" exitCode=137
Oct 30 01:52:14.990629 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:14.990138    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-4349cd"
Oct 30 01:52:14.990629 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:14.990184    4864 scope.go:122] "RemoveContainer" containerID="987cf56aee8f884391327f4a8fd25070a566c9506d72c262c51eb4face47eede"
Oct 30 01:52:15.000212 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:15.000186    4864 scope.go:122] "RemoveContainer" containerID="987cf56aee8f884391327f4a8fd25070a566c9506d72c262c51eb4face47eede"
Oct 30 01:52:15.000743 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:52:15.000499    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"987cf56aee8f884391327f4a8fd25070a566c9506d72c262c51eb4face47eede\": container with ID starting with 987cf56aee8f884391327f4a8fd25070a566c9506d72c262c51eb4face47eede not found: ID does not exist" containerID="987cf56aee8f884391327f4a8fd25070a566c9506d72c262c51eb4face47eede"
Oct 30 01:52:15.000743 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:15.000533    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"987cf56aee8f884391327f4a8fd25070a566c9506d72c262c51eb4face47eede"} err="failed to get container status \"987cf56aee8f884391327f4a8fd25070a566c9506d72c262c51eb4face47eede\": rpc error: code = NotFound desc = could not find container \"987cf56aee8f884391327f4a8fd25070a566c9506d72c262c51eb4face47eede\": container with ID starting with 987cf56aee8f884391327f4a8fd25070a566c9506d72c262c51eb4face47eede not found: ID does not exist"
Oct 30 01:52:15.012895 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:52:15.012865    4864 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7285-4349cd\" is forbidden: User \"system:node:ip-10-0-134-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-210.ec2.internal' and this object" podUID="6492d2c9-d332-4048-9bed-a904417ec3b3" pod="opendatahub/test-curl-7285-4349cd"
Oct 30 01:52:15.795358 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:15.795315    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6492d2c9-d332-4048-9bed-a904417ec3b3" path="/var/lib/kubelet/pods/6492d2c9-d332-4048-9bed-a904417ec3b3/volumes"
Oct 30 01:52:16.206403 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:16.206369    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-286c6e"
Oct 30 01:52:16.302629 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:16.302531    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3034998b-5403-4d4d-91e4-f9185beb21c5-kube-api-access-cmspf\" (UniqueName: \"kubernetes.io/projected/3034998b-5403-4d4d-91e4-f9185beb21c5-kube-api-access-cmspf\") pod \"3034998b-5403-4d4d-91e4-f9185beb21c5\" (UID: \"3034998b-5403-4d4d-91e4-f9185beb21c5\") "
Oct 30 01:52:16.304675 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:16.304645    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3034998b-5403-4d4d-91e4-f9185beb21c5-kube-api-access-cmspf" pod "3034998b-5403-4d4d-91e4-f9185beb21c5" (UID: "3034998b-5403-4d4d-91e4-f9185beb21c5"). InnerVolumeSpecName "kube-api-access-cmspf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:16.404682 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:16.404641    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmspf\" (UniqueName: \"kubernetes.io/projected/3034998b-5403-4d4d-91e4-f9185beb21c5-kube-api-access-cmspf\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:52:17.018095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:17.011051    4864 generic.go:356] "Generic (PLEG): container finished" podID="3034998b-5403-4d4d-91e4-f9185beb21c5" containerID="56c88e0061f72dbc4f6e69c1eb48da602841cf7673205ac5997dbe35ca549cf3" exitCode=137
Oct 30 01:52:17.018095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:17.011223    4864 scope.go:122] "RemoveContainer" containerID="56c88e0061f72dbc4f6e69c1eb48da602841cf7673205ac5997dbe35ca549cf3"
Oct 30 01:52:17.018095 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:17.011323    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-286c6e"
Oct 30 01:52:17.029384 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:17.029360    4864 scope.go:122] "RemoveContainer" containerID="56c88e0061f72dbc4f6e69c1eb48da602841cf7673205ac5997dbe35ca549cf3"
Oct 30 01:52:17.029674 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:52:17.029647    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c88e0061f72dbc4f6e69c1eb48da602841cf7673205ac5997dbe35ca549cf3\": container with ID starting with 56c88e0061f72dbc4f6e69c1eb48da602841cf7673205ac5997dbe35ca549cf3 not found: ID does not exist" containerID="56c88e0061f72dbc4f6e69c1eb48da602841cf7673205ac5997dbe35ca549cf3"
Oct 30 01:52:17.029742 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:17.029686    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c88e0061f72dbc4f6e69c1eb48da602841cf7673205ac5997dbe35ca549cf3"} err="failed to get container status \"56c88e0061f72dbc4f6e69c1eb48da602841cf7673205ac5997dbe35ca549cf3\": rpc error: code = NotFound desc = could not find container \"56c88e0061f72dbc4f6e69c1eb48da602841cf7673205ac5997dbe35ca549cf3\": container with ID starting with 56c88e0061f72dbc4f6e69c1eb48da602841cf7673205ac5997dbe35ca549cf3 not found: ID does not exist"
Oct 30 01:52:17.031829 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:52:17.031803    4864 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7285-286c6e\" is forbidden: User \"system:node:ip-10-0-134-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-210.ec2.internal' and this object" podUID="3034998b-5403-4d4d-91e4-f9185beb21c5" pod="opendatahub/test-curl-7285-286c6e"
Oct 30 01:52:17.796642 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:17.796610    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3034998b-5403-4d4d-91e4-f9185beb21c5" path="/var/lib/kubelet/pods/3034998b-5403-4d4d-91e4-f9185beb21c5/volumes"
Oct 30 01:52:17.801297 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:52:17.801273    4864 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7285-286c6e\" is forbidden: User \"system:node:ip-10-0-134-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-210.ec2.internal' and this object" podUID="3034998b-5403-4d4d-91e4-f9185beb21c5" pod="opendatahub/test-curl-7285-286c6e"
Oct 30 01:52:19.040658 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:52:19.040619    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:52:31.696736 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.696703    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"]
Oct 30 01:52:31.697388 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.697372    4864 cpu_manager.go:414] "containerMap: removing container" podUID="3034998b-5403-4d4d-91e4-f9185beb21c5" containerName="test-curl-7285-286c6e"
Oct 30 01:52:31.697441 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.697390    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3034998b-5403-4d4d-91e4-f9185beb21c5" containerName="test-curl-7285-286c6e"
Oct 30 01:52:31.697505 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.697496    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="3034998b-5403-4d4d-91e4-f9185beb21c5" containerName="test-curl-7285-286c6e"
Oct 30 01:52:31.702515 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.702496    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.717454 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.717434    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-8ae41e-openshift-default-dockercfg-22s5p\""
Oct 30 01:52:31.750189 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.750153    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"]
Oct 30 01:52:31.808580 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.808542    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-workload-socket\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.808753 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.808613    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-workload-certs\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.808753 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.808669    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-token\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.808829 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.808767    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-credential-socket\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.808829 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.808804    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istiod-ca-cert\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.808829 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.808824    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77fvv\" (UniqueName: \"kubernetes.io/projected/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-kube-api-access-77fvv\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.808963 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.808948    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-envoy\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.809020 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.809006    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-podinfo\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.809112 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.809098    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-data\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.910051 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.910016    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-credential-socket\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.910268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.910103    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istiod-ca-cert\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.910268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.910125    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77fvv\" (UniqueName: \"kubernetes.io/projected/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-kube-api-access-77fvv\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.910268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.910171    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-envoy\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.910268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.910244    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-podinfo\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.910496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.910319    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-data\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.910496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.910376    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-workload-socket\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.910496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.910407    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-workload-certs\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.910496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.910476    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-credential-socket\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.910715 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.910499    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-token\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.910774 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.910741    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-workload-socket\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.910774 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.910758    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-data\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.910881 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.910818    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-workload-certs\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.910944 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.910904    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istiod-ca-cert\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.912519 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.912496    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-envoy\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.912846 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.912825    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-podinfo\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.932689 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.932658    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-token\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:31.933433 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:31.933411    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77fvv\" (UniqueName: \"kubernetes.io/projected/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-kube-api-access-77fvv\") pod \"e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") " pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:32.015946 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:32.015913    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:32.333242 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:32.333212    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"]
Oct 30 01:52:32.334988 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:52:32.334953    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddfb64f0_d446_48ec_90e7_6f2c1e36920f.slice/crio-91d952f086a3d4a864708c81cb0338388e5a83a8fddfd10f9f426bcde669e314 WatchSource:0}: Error finding container 91d952f086a3d4a864708c81cb0338388e5a83a8fddfd10f9f426bcde669e314: Status 404 returned error can't find the container with id 91d952f086a3d4a864708c81cb0338388e5a83a8fddfd10f9f426bcde669e314
Oct 30 01:52:32.337724 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:32.337690    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:52:32.337820 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:32.337771    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:52:32.337880 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:32.337817    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:52:32.686711 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:32.686625    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"]
Oct 30 01:52:33.121641 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:33.121604    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz" event={"ID":"ddfb64f0-d446-48ec-90e7-6f2c1e36920f","Type":"ContainerStarted","Data":"0654d8e852974a70df6babbea8b16909d1bbd281129e449eb6819a656356c7d4"}
Oct 30 01:52:33.122101 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:33.121667    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz" event={"ID":"ddfb64f0-d446-48ec-90e7-6f2c1e36920f","Type":"ContainerStarted","Data":"91d952f086a3d4a864708c81cb0338388e5a83a8fddfd10f9f426bcde669e314"}
Oct 30 01:52:33.182299 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:33.182230    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz" podStartSLOduration=2.182211574 podStartE2EDuration="2.182211574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:33.18215067 +0000 UTC m=+4265.520252581" watchObservedRunningTime="2026-10-30 01:52:33.182211574 +0000 UTC m=+4265.520313485"
Oct 30 01:52:34.016086 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:34.016050    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:34.017926 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:34.017898    4864 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 01:52:34.018056 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:34.017957    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz" podUID="ddfb64f0-d446-48ec-90e7-6f2c1e36920f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 01:52:35.016419 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:35.016386    4864 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 01:52:35.016936 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:35.016443    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz" podUID="ddfb64f0-d446-48ec-90e7-6f2c1e36920f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 01:52:35.418016 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:35.417927    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"]
Oct 30 01:52:35.418231 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:35.418201    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7" podUID="afd0afda-48df-44ec-bebd-52768b7bdbc0" containerName="istio-proxy" containerID="cri-o://b6ffe50c10590157c17ab77d7f60c99261c3bf12b9c54086e9288dbacbb2d004" gracePeriod=30
Oct 30 01:52:36.016955 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:36.016917    4864 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 01:52:36.017332 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:36.016993    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz" podUID="ddfb64f0-d446-48ec-90e7-6f2c1e36920f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 01:52:36.822659 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:36.822623    4864 patch_prober.go:29] interesting pod/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:52:36.822835 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:36.822686    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7" podUID="afd0afda-48df-44ec-bebd-52768b7bdbc0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:52:37.016248 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:37.016209    4864 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 01:52:37.016451 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:37.016267    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz" podUID="ddfb64f0-d446-48ec-90e7-6f2c1e36920f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 01:52:38.016434 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:38.016397    4864 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 01:52:38.016823 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:38.016454    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz" podUID="ddfb64f0-d446-48ec-90e7-6f2c1e36920f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 01:52:39.124021 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:39.123989    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:39.124539 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:39.124514    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:39.125472 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:39.125444    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:52:40.687348 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.687288    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:52:40.771893 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.771846    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-envoy\") pod \"afd0afda-48df-44ec-bebd-52768b7bdbc0\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") "
Oct 30 01:52:40.772103 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.772009    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-workload-socket\") pod \"afd0afda-48df-44ec-bebd-52768b7bdbc0\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") "
Oct 30 01:52:40.772103 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.772067    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-workload-certs\") pod \"afd0afda-48df-44ec-bebd-52768b7bdbc0\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") "
Oct 30 01:52:40.772248 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.772147    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-podinfo\") pod \"afd0afda-48df-44ec-bebd-52768b7bdbc0\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") "
Oct 30 01:52:40.772248 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.772203    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-data\") pod \"afd0afda-48df-44ec-bebd-52768b7bdbc0\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") "
Oct 30 01:52:40.772376 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.772266    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd0afda-48df-44ec-bebd-52768b7bdbc0-kube-api-access-97vtg\" (UniqueName: \"kubernetes.io/projected/afd0afda-48df-44ec-bebd-52768b7bdbc0-kube-api-access-97vtg\") pod \"afd0afda-48df-44ec-bebd-52768b7bdbc0\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") "
Oct 30 01:52:40.772376 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.772330    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afd0afda-48df-44ec-bebd-52768b7bdbc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd0afda-48df-44ec-bebd-52768b7bdbc0-istiod-ca-cert\") pod \"afd0afda-48df-44ec-bebd-52768b7bdbc0\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") "
Oct 30 01:52:40.772376 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.772338    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-workload-socket" pod "afd0afda-48df-44ec-bebd-52768b7bdbc0" (UID: "afd0afda-48df-44ec-bebd-52768b7bdbc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:52:40.772376 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.772360    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-token\" (UniqueName: \"kubernetes.io/projected/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-token\") pod \"afd0afda-48df-44ec-bebd-52768b7bdbc0\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") "
Oct 30 01:52:40.772600 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.772396    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-credential-socket\") pod \"afd0afda-48df-44ec-bebd-52768b7bdbc0\" (UID: \"afd0afda-48df-44ec-bebd-52768b7bdbc0\") "
Oct 30 01:52:40.772929 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.772898    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-data" pod "afd0afda-48df-44ec-bebd-52768b7bdbc0" (UID: "afd0afda-48df-44ec-bebd-52768b7bdbc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:52:40.773065 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.772992    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:52:40.773065 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.773012    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:52:40.773283 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.773260    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-credential-socket" pod "afd0afda-48df-44ec-bebd-52768b7bdbc0" (UID: "afd0afda-48df-44ec-bebd-52768b7bdbc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:52:40.773590 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.773567    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd0afda-48df-44ec-bebd-52768b7bdbc0-istiod-ca-cert" pod "afd0afda-48df-44ec-bebd-52768b7bdbc0" (UID: "afd0afda-48df-44ec-bebd-52768b7bdbc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:52:40.773838 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.773816    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-workload-certs" pod "afd0afda-48df-44ec-bebd-52768b7bdbc0" (UID: "afd0afda-48df-44ec-bebd-52768b7bdbc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:52:40.774770 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.774743    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-envoy" pod "afd0afda-48df-44ec-bebd-52768b7bdbc0" (UID: "afd0afda-48df-44ec-bebd-52768b7bdbc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:52:40.774898 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.774872    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-podinfo" pod "afd0afda-48df-44ec-bebd-52768b7bdbc0" (UID: "afd0afda-48df-44ec-bebd-52768b7bdbc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:52:40.775479 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.775447    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd0afda-48df-44ec-bebd-52768b7bdbc0-kube-api-access-97vtg" pod "afd0afda-48df-44ec-bebd-52768b7bdbc0" (UID: "afd0afda-48df-44ec-bebd-52768b7bdbc0"). InnerVolumeSpecName "kube-api-access-97vtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:40.776687 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.776662    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-token" pod "afd0afda-48df-44ec-bebd-52768b7bdbc0" (UID: "afd0afda-48df-44ec-bebd-52768b7bdbc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:40.874159 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.874118    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:52:40.874159 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.874151    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:52:40.874365 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.874171    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97vtg\" (UniqueName: \"kubernetes.io/projected/afd0afda-48df-44ec-bebd-52768b7bdbc0-kube-api-access-97vtg\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:52:40.874365 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.874180    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd0afda-48df-44ec-bebd-52768b7bdbc0-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:52:40.874365 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.874189    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:52:40.874365 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.874196    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:52:40.874365 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:40.874204    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd0afda-48df-44ec-bebd-52768b7bdbc0-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:52:41.178664 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:41.178628    4864 generic.go:356] "Generic (PLEG): container finished" podID="afd0afda-48df-44ec-bebd-52768b7bdbc0" containerID="b6ffe50c10590157c17ab77d7f60c99261c3bf12b9c54086e9288dbacbb2d004" exitCode=0
Oct 30 01:52:41.178868 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:41.178672    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"
Oct 30 01:52:41.178868 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:41.178723    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7" event={"ID":"afd0afda-48df-44ec-bebd-52768b7bdbc0","Type":"ContainerDied","Data":"b6ffe50c10590157c17ab77d7f60c99261c3bf12b9c54086e9288dbacbb2d004"}
Oct 30 01:52:41.178868 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:41.178777    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7" event={"ID":"afd0afda-48df-44ec-bebd-52768b7bdbc0","Type":"ContainerDied","Data":"363684e05dcaaf50a4ece9278e9fde3e5324e871b829069d18c8295f190da86b"}
Oct 30 01:52:41.178868 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:41.178806    4864 scope.go:122] "RemoveContainer" containerID="b6ffe50c10590157c17ab77d7f60c99261c3bf12b9c54086e9288dbacbb2d004"
Oct 30 01:52:41.189694 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:41.189671    4864 scope.go:122] "RemoveContainer" containerID="9af37006555e0a21547d34f8b2eff55428ddccc6df1acbf519db4b089712b7dc"
Oct 30 01:52:41.198975 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:41.198958    4864 scope.go:122] "RemoveContainer" containerID="b6ffe50c10590157c17ab77d7f60c99261c3bf12b9c54086e9288dbacbb2d004"
Oct 30 01:52:41.199271 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:52:41.199248    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6ffe50c10590157c17ab77d7f60c99261c3bf12b9c54086e9288dbacbb2d004\": container with ID starting with b6ffe50c10590157c17ab77d7f60c99261c3bf12b9c54086e9288dbacbb2d004 not found: ID does not exist" containerID="b6ffe50c10590157c17ab77d7f60c99261c3bf12b9c54086e9288dbacbb2d004"
Oct 30 01:52:41.199342 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:41.199277    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6ffe50c10590157c17ab77d7f60c99261c3bf12b9c54086e9288dbacbb2d004"} err="failed to get container status \"b6ffe50c10590157c17ab77d7f60c99261c3bf12b9c54086e9288dbacbb2d004\": rpc error: code = NotFound desc = could not find container \"b6ffe50c10590157c17ab77d7f60c99261c3bf12b9c54086e9288dbacbb2d004\": container with ID starting with b6ffe50c10590157c17ab77d7f60c99261c3bf12b9c54086e9288dbacbb2d004 not found: ID does not exist"
Oct 30 01:52:41.199342 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:41.199294    4864 scope.go:122] "RemoveContainer" containerID="9af37006555e0a21547d34f8b2eff55428ddccc6df1acbf519db4b089712b7dc"
Oct 30 01:52:41.199551 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:52:41.199531    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af37006555e0a21547d34f8b2eff55428ddccc6df1acbf519db4b089712b7dc\": container with ID starting with 9af37006555e0a21547d34f8b2eff55428ddccc6df1acbf519db4b089712b7dc not found: ID does not exist" containerID="9af37006555e0a21547d34f8b2eff55428ddccc6df1acbf519db4b089712b7dc"
Oct 30 01:52:41.199593 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:41.199557    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af37006555e0a21547d34f8b2eff55428ddccc6df1acbf519db4b089712b7dc"} err="failed to get container status \"9af37006555e0a21547d34f8b2eff55428ddccc6df1acbf519db4b089712b7dc\": rpc error: code = NotFound desc = could not find container \"9af37006555e0a21547d34f8b2eff55428ddccc6df1acbf519db4b089712b7dc\": container with ID starting with 9af37006555e0a21547d34f8b2eff55428ddccc6df1acbf519db4b089712b7dc not found: ID does not exist"
Oct 30 01:52:41.236098 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:41.236056    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"]
Oct 30 01:52:41.246059 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:41.246037    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-94ea1e-openshift-default-5b75447d77-bvfr7"]
Oct 30 01:52:41.794400 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:41.794364    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afd0afda-48df-44ec-bebd-52768b7bdbc0" path="/var/lib/kubelet/pods/afd0afda-48df-44ec-bebd-52768b7bdbc0/volumes"
Oct 30 01:52:47.464592 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:47.464557    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n"]
Oct 30 01:52:47.709371 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:47.709330    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj" podUID="a87828aa-8038-4643-834a-bbdb78432874" containerName="main" containerID="cri-o://d1cfd16463abd2603a3cbb6ba56e277eeea77e3af36678edae7cf1344fdaf390" gracePeriod=45
Oct 30 01:52:48.010611 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.010582    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:52:48.071870 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.071840    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-dshm\") pod \"a87828aa-8038-4643-834a-bbdb78432874\" (UID: \"a87828aa-8038-4643-834a-bbdb78432874\") "
Oct 30 01:52:48.072089 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.072064    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-model-cache\") pod \"a87828aa-8038-4643-834a-bbdb78432874\" (UID: \"a87828aa-8038-4643-834a-bbdb78432874\") "
Oct 30 01:52:48.072178 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.072108    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a87828aa-8038-4643-834a-bbdb78432874-tls-certs\" (UniqueName: \"kubernetes.io/secret/a87828aa-8038-4643-834a-bbdb78432874-tls-certs\") pod \"a87828aa-8038-4643-834a-bbdb78432874\" (UID: \"a87828aa-8038-4643-834a-bbdb78432874\") "
Oct 30 01:52:48.072178 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.072138    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-tmp-dir\") pod \"a87828aa-8038-4643-834a-bbdb78432874\" (UID: \"a87828aa-8038-4643-834a-bbdb78432874\") "
Oct 30 01:52:48.072178 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.072170    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a87828aa-8038-4643-834a-bbdb78432874-kube-api-access-jstwt\" (UniqueName: \"kubernetes.io/projected/a87828aa-8038-4643-834a-bbdb78432874-kube-api-access-jstwt\") pod \"a87828aa-8038-4643-834a-bbdb78432874\" (UID: \"a87828aa-8038-4643-834a-bbdb78432874\") "
Oct 30 01:52:48.072357 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.072262    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-home\" (UniqueName: \"kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-home\") pod \"a87828aa-8038-4643-834a-bbdb78432874\" (UID: \"a87828aa-8038-4643-834a-bbdb78432874\") "
Oct 30 01:52:48.072420 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.072352    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-model-cache" pod "a87828aa-8038-4643-834a-bbdb78432874" (UID: "a87828aa-8038-4643-834a-bbdb78432874"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:52:48.072475 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.072418    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-tmp-dir" pod "a87828aa-8038-4643-834a-bbdb78432874" (UID: "a87828aa-8038-4643-834a-bbdb78432874"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:52:48.072642 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.072621    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-home" pod "a87828aa-8038-4643-834a-bbdb78432874" (UID: "a87828aa-8038-4643-834a-bbdb78432874"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:52:48.072944 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.072921    4864 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-model-cache\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:52:48.073125 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.073110    4864 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-tmp-dir\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:52:48.073251 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.073230    4864 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-home\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:52:48.074716 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.074695    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a87828aa-8038-4643-834a-bbdb78432874-kube-api-access-jstwt" pod "a87828aa-8038-4643-834a-bbdb78432874" (UID: "a87828aa-8038-4643-834a-bbdb78432874"). InnerVolumeSpecName "kube-api-access-jstwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:48.074839 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.074822    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-dshm" pod "a87828aa-8038-4643-834a-bbdb78432874" (UID: "a87828aa-8038-4643-834a-bbdb78432874"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:52:48.078280 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.078260    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a87828aa-8038-4643-834a-bbdb78432874-tls-certs" pod "a87828aa-8038-4643-834a-bbdb78432874" (UID: "a87828aa-8038-4643-834a-bbdb78432874"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:52:48.174426 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.174395    4864 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a87828aa-8038-4643-834a-bbdb78432874-tls-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:52:48.174426 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.174422    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jstwt\" (UniqueName: \"kubernetes.io/projected/a87828aa-8038-4643-834a-bbdb78432874-kube-api-access-jstwt\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:52:48.174426 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.174432    4864 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a87828aa-8038-4643-834a-bbdb78432874-dshm\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:52:48.259422 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.259317    4864 generic.go:356] "Generic (PLEG): container finished" podID="a87828aa-8038-4643-834a-bbdb78432874" containerID="d1cfd16463abd2603a3cbb6ba56e277eeea77e3af36678edae7cf1344fdaf390" exitCode=0
Oct 30 01:52:48.259669 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.259647    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj" event={"ID":"a87828aa-8038-4643-834a-bbdb78432874","Type":"ContainerDied","Data":"d1cfd16463abd2603a3cbb6ba56e277eeea77e3af36678edae7cf1344fdaf390"}
Oct 30 01:52:48.259753 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.259694    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj" event={"ID":"a87828aa-8038-4643-834a-bbdb78432874","Type":"ContainerDied","Data":"9e9c578f6bdf58e5916773a66f558e041ac9891f0d8a31405853d0402a3bc3ee"}
Oct 30 01:52:48.259753 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.259723    4864 scope.go:122] "RemoveContainer" containerID="d1cfd16463abd2603a3cbb6ba56e277eeea77e3af36678edae7cf1344fdaf390"
Oct 30 01:52:48.259914 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.259897    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"
Oct 30 01:52:48.280947 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.280926    4864 scope.go:122] "RemoveContainer" containerID="d1cfd16463abd2603a3cbb6ba56e277eeea77e3af36678edae7cf1344fdaf390"
Oct 30 01:52:48.281323 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:52:48.281297    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1cfd16463abd2603a3cbb6ba56e277eeea77e3af36678edae7cf1344fdaf390\": container with ID starting with d1cfd16463abd2603a3cbb6ba56e277eeea77e3af36678edae7cf1344fdaf390 not found: ID does not exist" containerID="d1cfd16463abd2603a3cbb6ba56e277eeea77e3af36678edae7cf1344fdaf390"
Oct 30 01:52:48.281411 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.281332    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1cfd16463abd2603a3cbb6ba56e277eeea77e3af36678edae7cf1344fdaf390"} err="failed to get container status \"d1cfd16463abd2603a3cbb6ba56e277eeea77e3af36678edae7cf1344fdaf390\": rpc error: code = NotFound desc = could not find container \"d1cfd16463abd2603a3cbb6ba56e277eeea77e3af36678edae7cf1344fdaf390\": container with ID starting with d1cfd16463abd2603a3cbb6ba56e277eeea77e3af36678edae7cf1344fdaf390 not found: ID does not exist"
Oct 30 01:52:48.310435 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.310403    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"]
Oct 30 01:52:48.316599 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:48.316572    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-35042a/rate-test-model-35042a-kserve-7f59b44fd5-vk4sj"]
Oct 30 01:52:49.040463 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:52:49.040427    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:52:49.793922 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:49.793879    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a87828aa-8038-4643-834a-bbdb78432874" path="/var/lib/kubelet/pods/a87828aa-8038-4643-834a-bbdb78432874/volumes"
Oct 30 01:52:50.732827 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:50.732792    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"]
Oct 30 01:52:50.733642 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:50.733608    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp" podUID="ac76439d-8c30-4810-b242-1a3e1ed2b556" containerName="maas-api" containerID="cri-o://21a8e27de431d5693ff1ec193b81ac9ec47e372cad70a9f3d354206048e8d87f" gracePeriod=30
Oct 30 01:52:51.024133 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:51.024067    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"
Oct 30 01:52:51.110522 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:51.110488    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac76439d-8c30-4810-b242-1a3e1ed2b556-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac76439d-8c30-4810-b242-1a3e1ed2b556-openshift-service-ca\") pod \"ac76439d-8c30-4810-b242-1a3e1ed2b556\" (UID: \"ac76439d-8c30-4810-b242-1a3e1ed2b556\") "
Oct 30 01:52:51.110696 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:51.110578    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac76439d-8c30-4810-b242-1a3e1ed2b556-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac76439d-8c30-4810-b242-1a3e1ed2b556-maas-api-tls\") pod \"ac76439d-8c30-4810-b242-1a3e1ed2b556\" (UID: \"ac76439d-8c30-4810-b242-1a3e1ed2b556\") "
Oct 30 01:52:51.110696 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:51.110640    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac76439d-8c30-4810-b242-1a3e1ed2b556-kube-api-access-hml27\" (UniqueName: \"kubernetes.io/projected/ac76439d-8c30-4810-b242-1a3e1ed2b556-kube-api-access-hml27\") pod \"ac76439d-8c30-4810-b242-1a3e1ed2b556\" (UID: \"ac76439d-8c30-4810-b242-1a3e1ed2b556\") "
Oct 30 01:52:51.110848 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:51.110777    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac76439d-8c30-4810-b242-1a3e1ed2b556-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac76439d-8c30-4810-b242-1a3e1ed2b556-metrics-certs\") pod \"ac76439d-8c30-4810-b242-1a3e1ed2b556\" (UID: \"ac76439d-8c30-4810-b242-1a3e1ed2b556\") "
Oct 30 01:52:51.111255 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:51.111153    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac76439d-8c30-4810-b242-1a3e1ed2b556-openshift-service-ca" pod "ac76439d-8c30-4810-b242-1a3e1ed2b556" (UID: "ac76439d-8c30-4810-b242-1a3e1ed2b556"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:52:51.111506 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:51.111482    4864 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac76439d-8c30-4810-b242-1a3e1ed2b556-openshift-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:52:51.114403 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:51.114356    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac76439d-8c30-4810-b242-1a3e1ed2b556-kube-api-access-hml27" pod "ac76439d-8c30-4810-b242-1a3e1ed2b556" (UID: "ac76439d-8c30-4810-b242-1a3e1ed2b556"). InnerVolumeSpecName "kube-api-access-hml27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:51.114632 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:51.114600    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac76439d-8c30-4810-b242-1a3e1ed2b556-maas-api-tls" pod "ac76439d-8c30-4810-b242-1a3e1ed2b556" (UID: "ac76439d-8c30-4810-b242-1a3e1ed2b556"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:52:51.116168 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:51.116129    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac76439d-8c30-4810-b242-1a3e1ed2b556-metrics-certs" pod "ac76439d-8c30-4810-b242-1a3e1ed2b556" (UID: "ac76439d-8c30-4810-b242-1a3e1ed2b556"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:52:51.215869 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:51.215828    4864 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac76439d-8c30-4810-b242-1a3e1ed2b556-metrics-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:52:51.215869 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:51.215868    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac76439d-8c30-4810-b242-1a3e1ed2b556-maas-api-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:52:51.216327 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:51.215883    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hml27\" (UniqueName: \"kubernetes.io/projected/ac76439d-8c30-4810-b242-1a3e1ed2b556-kube-api-access-hml27\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:52:51.283224 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:51.283136    4864 generic.go:356] "Generic (PLEG): container finished" podID="ac76439d-8c30-4810-b242-1a3e1ed2b556" containerID="21a8e27de431d5693ff1ec193b81ac9ec47e372cad70a9f3d354206048e8d87f" exitCode=0
Oct 30 01:52:51.283224 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:51.283184    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"
Oct 30 01:52:51.283224 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:51.283188    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp" event={"ID":"ac76439d-8c30-4810-b242-1a3e1ed2b556","Type":"ContainerDied","Data":"21a8e27de431d5693ff1ec193b81ac9ec47e372cad70a9f3d354206048e8d87f"}
Oct 30 01:52:51.283498 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:51.283254    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp" event={"ID":"ac76439d-8c30-4810-b242-1a3e1ed2b556","Type":"ContainerDied","Data":"9a08d0667ac5837f4dc6116f2186ea3f9f887c27c072aed5719e7580b0abfd10"}
Oct 30 01:52:51.283498 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:51.283287    4864 scope.go:122] "RemoveContainer" containerID="21a8e27de431d5693ff1ec193b81ac9ec47e372cad70a9f3d354206048e8d87f"
Oct 30 01:52:51.306341 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:51.300246    4864 scope.go:122] "RemoveContainer" containerID="21a8e27de431d5693ff1ec193b81ac9ec47e372cad70a9f3d354206048e8d87f"
Oct 30 01:52:51.306341 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:52:51.300723    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a8e27de431d5693ff1ec193b81ac9ec47e372cad70a9f3d354206048e8d87f\": container with ID starting with 21a8e27de431d5693ff1ec193b81ac9ec47e372cad70a9f3d354206048e8d87f not found: ID does not exist" containerID="21a8e27de431d5693ff1ec193b81ac9ec47e372cad70a9f3d354206048e8d87f"
Oct 30 01:52:51.306341 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:51.300759    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a8e27de431d5693ff1ec193b81ac9ec47e372cad70a9f3d354206048e8d87f"} err="failed to get container status \"21a8e27de431d5693ff1ec193b81ac9ec47e372cad70a9f3d354206048e8d87f\": rpc error: code = NotFound desc = could not find container \"21a8e27de431d5693ff1ec193b81ac9ec47e372cad70a9f3d354206048e8d87f\": container with ID starting with 21a8e27de431d5693ff1ec193b81ac9ec47e372cad70a9f3d354206048e8d87f not found: ID does not exist"
Oct 30 01:52:51.335121 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:51.335092    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"]
Oct 30 01:52:51.339892 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:51.339865    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-499482-8585c74d7b-ldhvp"]
Oct 30 01:52:51.794728 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:51.794681    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac76439d-8c30-4810-b242-1a3e1ed2b556" path="/var/lib/kubelet/pods/ac76439d-8c30-4810-b242-1a3e1ed2b556/volumes"
Oct 30 01:52:52.467937 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:52.467858    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n" podUID="d0822059-3672-4595-967c-0796222fd966" containerName="payload-pre-processing" containerID="cri-o://b90616317662e8bc1eaa935b9f27ae9a6a54e8b9035aa164a9c384f3c83b9a2c" gracePeriod=25
Oct 30 01:52:52.754025 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:52.753993    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n"
Oct 30 01:52:52.837646 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:52.837613    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0822059-3672-4595-967c-0796222fd966-kube-api-access-shkq9\" (UniqueName: \"kubernetes.io/projected/d0822059-3672-4595-967c-0796222fd966-kube-api-access-shkq9\") pod \"d0822059-3672-4595-967c-0796222fd966\" (UID: \"d0822059-3672-4595-967c-0796222fd966\") "
Oct 30 01:52:52.838035 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:52.837874    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0822059-3672-4595-967c-0796222fd966-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0822059-3672-4595-967c-0796222fd966-plugins-config-volume\") pod \"d0822059-3672-4595-967c-0796222fd966\" (UID: \"d0822059-3672-4595-967c-0796222fd966\") "
Oct 30 01:52:52.838487 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:52.838344    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0822059-3672-4595-967c-0796222fd966-plugins-config-volume" pod "d0822059-3672-4595-967c-0796222fd966" (UID: "d0822059-3672-4595-967c-0796222fd966"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:52:52.840539 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:52.840501    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0822059-3672-4595-967c-0796222fd966-kube-api-access-shkq9" pod "d0822059-3672-4595-967c-0796222fd966" (UID: "d0822059-3672-4595-967c-0796222fd966"). InnerVolumeSpecName "kube-api-access-shkq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:52.939490 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:52.939454    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shkq9\" (UniqueName: \"kubernetes.io/projected/d0822059-3672-4595-967c-0796222fd966-kube-api-access-shkq9\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:52:52.939490 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:52.939485    4864 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0822059-3672-4595-967c-0796222fd966-plugins-config-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:52:53.297948 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:53.297913    4864 generic.go:356] "Generic (PLEG): container finished" podID="d0822059-3672-4595-967c-0796222fd966" containerID="b90616317662e8bc1eaa935b9f27ae9a6a54e8b9035aa164a9c384f3c83b9a2c" exitCode=0
Oct 30 01:52:53.298149 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:53.297953    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n"
Oct 30 01:52:53.298149 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:53.297982    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n" event={"ID":"d0822059-3672-4595-967c-0796222fd966","Type":"ContainerDied","Data":"b90616317662e8bc1eaa935b9f27ae9a6a54e8b9035aa164a9c384f3c83b9a2c"}
Oct 30 01:52:53.298149 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:53.298026    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n" event={"ID":"d0822059-3672-4595-967c-0796222fd966","Type":"ContainerDied","Data":"f87d57e1ce1043cb49d2b6100e2157aaf97a68f2073b2adaa9e2a435744299bc"}
Oct 30 01:52:53.298149 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:53.298050    4864 scope.go:122] "RemoveContainer" containerID="b90616317662e8bc1eaa935b9f27ae9a6a54e8b9035aa164a9c384f3c83b9a2c"
Oct 30 01:52:53.308344 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:53.308326    4864 scope.go:122] "RemoveContainer" containerID="b90616317662e8bc1eaa935b9f27ae9a6a54e8b9035aa164a9c384f3c83b9a2c"
Oct 30 01:52:53.308632 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:52:53.308608    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b90616317662e8bc1eaa935b9f27ae9a6a54e8b9035aa164a9c384f3c83b9a2c\": container with ID starting with b90616317662e8bc1eaa935b9f27ae9a6a54e8b9035aa164a9c384f3c83b9a2c not found: ID does not exist" containerID="b90616317662e8bc1eaa935b9f27ae9a6a54e8b9035aa164a9c384f3c83b9a2c"
Oct 30 01:52:53.308732 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:53.308638    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b90616317662e8bc1eaa935b9f27ae9a6a54e8b9035aa164a9c384f3c83b9a2c"} err="failed to get container status \"b90616317662e8bc1eaa935b9f27ae9a6a54e8b9035aa164a9c384f3c83b9a2c\": rpc error: code = NotFound desc = could not find container \"b90616317662e8bc1eaa935b9f27ae9a6a54e8b9035aa164a9c384f3c83b9a2c\": container with ID starting with b90616317662e8bc1eaa935b9f27ae9a6a54e8b9035aa164a9c384f3c83b9a2c not found: ID does not exist"
Oct 30 01:52:53.344271 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:53.344242    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n"]
Oct 30 01:52:53.354297 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:53.354273    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-df4db1-5ccf55bc99-v7w2n"]
Oct 30 01:52:53.794429 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:53.794390    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0822059-3672-4595-967c-0796222fd966" path="/var/lib/kubelet/pods/d0822059-3672-4595-967c-0796222fd966/volumes"
Oct 30 01:52:59.192019 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:52:59.191986    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"]
Oct 30 01:53:14.213988 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:14.213943    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25" podUID="55dbdf6b-c1f2-4e0a-907a-7cd59c30e989" containerName="main" containerID="cri-o://79e76a1a8f18ff1b40faa2390aceb61c12ff600ab53bbac4a3ccaa654239f6cf" gracePeriod=45
Oct 30 01:53:14.489554 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:14.489519    4864 generic.go:356] "Generic (PLEG): container finished" podID="55dbdf6b-c1f2-4e0a-907a-7cd59c30e989" containerID="79e76a1a8f18ff1b40faa2390aceb61c12ff600ab53bbac4a3ccaa654239f6cf" exitCode=0
Oct 30 01:53:14.489709 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:14.489590    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25" event={"ID":"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989","Type":"ContainerDied","Data":"79e76a1a8f18ff1b40faa2390aceb61c12ff600ab53bbac4a3ccaa654239f6cf"}
Oct 30 01:53:14.489709 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:14.489633    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25" event={"ID":"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989","Type":"ContainerDied","Data":"ce3b8957a1f0e44288ac0597d8575f4ca283417efe83a98d1316782533b3359d"}
Oct 30 01:53:14.489709 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:14.489646    4864 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce3b8957a1f0e44288ac0597d8575f4ca283417efe83a98d1316782533b3359d"
Oct 30 01:53:14.509937 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:14.509903    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:53:14.635567 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:14.635536    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-home\" (UniqueName: \"kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-home\") pod \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\" (UID: \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\") "
Oct 30 01:53:14.635737 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:14.635603    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-dshm\" (UniqueName: \"kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-dshm\") pod \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\" (UID: \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\") "
Oct 30 01:53:14.635737 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:14.635720    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-model-cache\") pod \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\" (UID: \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\") "
Oct 30 01:53:14.635842 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:14.635757    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-kube-api-access-ft2fk\" (UniqueName: \"kubernetes.io/projected/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-kube-api-access-ft2fk\") pod \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\" (UID: \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\") "
Oct 30 01:53:14.635842 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:14.635783    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-home" pod "55dbdf6b-c1f2-4e0a-907a-7cd59c30e989" (UID: "55dbdf6b-c1f2-4e0a-907a-7cd59c30e989"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:14.635928 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:14.635852    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-tmp-dir\") pod \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\" (UID: \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\") "
Oct 30 01:53:14.635928 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:14.635917    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-tls-certs\" (UniqueName: \"kubernetes.io/secret/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-tls-certs\") pod \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\" (UID: \"55dbdf6b-c1f2-4e0a-907a-7cd59c30e989\") "
Oct 30 01:53:14.636043 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:14.635998    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-model-cache" pod "55dbdf6b-c1f2-4e0a-907a-7cd59c30e989" (UID: "55dbdf6b-c1f2-4e0a-907a-7cd59c30e989"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:14.636206 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:14.636185    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-tmp-dir" pod "55dbdf6b-c1f2-4e0a-907a-7cd59c30e989" (UID: "55dbdf6b-c1f2-4e0a-907a-7cd59c30e989"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:14.636500 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:14.636481    4864 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-tmp-dir\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:14.636583 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:14.636505    4864 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-home\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:14.636583 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:14.636519    4864 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-model-cache\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:14.637910 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:14.637878    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-tls-certs" pod "55dbdf6b-c1f2-4e0a-907a-7cd59c30e989" (UID: "55dbdf6b-c1f2-4e0a-907a-7cd59c30e989"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:53:14.638015 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:14.637943    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-dshm" pod "55dbdf6b-c1f2-4e0a-907a-7cd59c30e989" (UID: "55dbdf6b-c1f2-4e0a-907a-7cd59c30e989"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:14.638015 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:14.637963    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-kube-api-access-ft2fk" pod "55dbdf6b-c1f2-4e0a-907a-7cd59c30e989" (UID: "55dbdf6b-c1f2-4e0a-907a-7cd59c30e989"). InnerVolumeSpecName "kube-api-access-ft2fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:14.737733 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:14.737658    4864 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-tls-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:14.737733 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:14.737684    4864 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-dshm\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:14.737733 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:14.737693    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft2fk\" (UniqueName: \"kubernetes.io/projected/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989-kube-api-access-ft2fk\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:15.496790 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:15.496756    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"
Oct 30 01:53:15.543514 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:15.543478    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"]
Oct 30 01:53:15.553600 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:15.553572    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8ae41e/sub-test-model-8ae41e-kserve-679b595c85-qkn25"]
Oct 30 01:53:15.794338 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:15.794260    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55dbdf6b-c1f2-4e0a-907a-7cd59c30e989" path="/var/lib/kubelet/pods/55dbdf6b-c1f2-4e0a-907a-7cd59c30e989/volumes"
Oct 30 01:53:19.042500 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:53:19.042466    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:53:39.083697 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:39.083662    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"]
Oct 30 01:53:39.084246 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:39.083881    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx" podUID="bc75808d-fb11-4820-939a-18037b0c2791" containerName="istio-proxy" containerID="cri-o://49b43214c8922cf236068aa83328006e70e5f4b5d6c9a6d70a55803393cf0e2f" gracePeriod=30
Oct 30 01:53:39.094607 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:39.094581    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"]
Oct 30 01:53:39.094787 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:39.094762    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t" podUID="89b1105d-0395-4e80-a63b-7e8e0cce276d" containerName="istio-proxy" containerID="cri-o://ca24b5ad03af3279d18cb6f628f6947a3928918cf5028902146eaa2e6c999ce1" gracePeriod=30
Oct 30 01:53:39.097546 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:39.097518    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"]
Oct 30 01:53:39.097711 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:39.097689    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s" podUID="e92bdf87-b7d6-4060-9615-70ae77214bc0" containerName="istio-proxy" containerID="cri-o://f87e5624c1595c3e1095260cb5fa030036ab26d533f8eaddf333d1a776c477a6" gracePeriod=30
Oct 30 01:53:39.103879 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:39.103854    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"]
Oct 30 01:53:39.104110 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:39.104045    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld" podUID="9de67abe-4f7c-441f-9e3e-cf34c74068d9" containerName="istio-proxy" containerID="cri-o://fa4d3a238a1c681ec10393a53e58be3242307bd7f9be38200f339e40cec41164" gracePeriod=30
Oct 30 01:53:41.648622 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:41.648582    4864 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zct5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:53:41.649010 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:41.648643    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s" podUID="e92bdf87-b7d6-4060-9615-70ae77214bc0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:53:44.397308 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.397274    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:53:44.431493 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.431468    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:53:44.480555 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.480526    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:53:44.486746 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.486723    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:53:44.572848 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.572807    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istiod-ca-cert\") pod \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") "
Oct 30 01:53:44.572848 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.572848    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-workload-certs\") pod \"e92bdf87-b7d6-4060-9615-70ae77214bc0\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") "
Oct 30 01:53:44.573276 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.572879    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e92bdf87-b7d6-4060-9615-70ae77214bc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92bdf87-b7d6-4060-9615-70ae77214bc0-istiod-ca-cert\") pod \"e92bdf87-b7d6-4060-9615-70ae77214bc0\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") "
Oct 30 01:53:44.573276 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.572991    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-credential-socket\") pod \"e92bdf87-b7d6-4060-9615-70ae77214bc0\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") "
Oct 30 01:53:44.573276 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573041    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9de67abe-4f7c-441f-9e3e-cf34c74068d9-kube-api-access-7f9vf\" (UniqueName: \"kubernetes.io/projected/9de67abe-4f7c-441f-9e3e-cf34c74068d9-kube-api-access-7f9vf\") pod \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") "
Oct 30 01:53:44.573276 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573091    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-credential-socket\") pod \"89b1105d-0395-4e80-a63b-7e8e0cce276d\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") "
Oct 30 01:53:44.573276 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573122    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-credential-socket\") pod \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") "
Oct 30 01:53:44.573276 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573160    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-data\") pod \"89b1105d-0395-4e80-a63b-7e8e0cce276d\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") "
Oct 30 01:53:44.573276 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573163    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-workload-certs" pod "e92bdf87-b7d6-4060-9615-70ae77214bc0" (UID: "e92bdf87-b7d6-4060-9615-70ae77214bc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.573276 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573210    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-data\") pod \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") "
Oct 30 01:53:44.573276 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573240    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istiod-ca-cert" pod "9de67abe-4f7c-441f-9e3e-cf34c74068d9" (UID: "9de67abe-4f7c-441f-9e3e-cf34c74068d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:53:44.573276 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573247    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92bdf87-b7d6-4060-9615-70ae77214bc0-istiod-ca-cert" pod "e92bdf87-b7d6-4060-9615-70ae77214bc0" (UID: "e92bdf87-b7d6-4060-9615-70ae77214bc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:53:44.573276 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573258    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc75808d-fb11-4820-939a-18037b0c2791-kube-api-access-zj9zf\" (UniqueName: \"kubernetes.io/projected/bc75808d-fb11-4820-939a-18037b0c2791-kube-api-access-zj9zf\") pod \"bc75808d-fb11-4820-939a-18037b0c2791\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") "
Oct 30 01:53:44.573890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573298    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc75808d-fb11-4820-939a-18037b0c2791-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc75808d-fb11-4820-939a-18037b0c2791-istio-podinfo\") pod \"bc75808d-fb11-4820-939a-18037b0c2791\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") "
Oct 30 01:53:44.573890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573333    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-token\" (UniqueName: \"kubernetes.io/projected/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-token\") pod \"e92bdf87-b7d6-4060-9615-70ae77214bc0\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") "
Oct 30 01:53:44.573890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573366    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-workload-certs\") pod \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") "
Oct 30 01:53:44.573890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573397    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-workload-socket\") pod \"89b1105d-0395-4e80-a63b-7e8e0cce276d\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") "
Oct 30 01:53:44.573890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573427    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-token\" (UniqueName: \"kubernetes.io/projected/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-token\") pod \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") "
Oct 30 01:53:44.573890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573460    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-workload-socket\") pod \"e92bdf87-b7d6-4060-9615-70ae77214bc0\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") "
Oct 30 01:53:44.573890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573506    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-credential-socket" pod "e92bdf87-b7d6-4060-9615-70ae77214bc0" (UID: "e92bdf87-b7d6-4060-9615-70ae77214bc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.573890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573516    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-podinfo\") pod \"89b1105d-0395-4e80-a63b-7e8e0cce276d\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") "
Oct 30 01:53:44.573890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573518    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-credential-socket" pod "9de67abe-4f7c-441f-9e3e-cf34c74068d9" (UID: "9de67abe-4f7c-441f-9e3e-cf34c74068d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.573890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573589    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-workload-socket\") pod \"bc75808d-fb11-4820-939a-18037b0c2791\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") "
Oct 30 01:53:44.573890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573636    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-credential-socket\") pod \"bc75808d-fb11-4820-939a-18037b0c2791\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") "
Oct 30 01:53:44.573890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573674    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-istio-data\") pod \"bc75808d-fb11-4820-939a-18037b0c2791\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") "
Oct 30 01:53:44.573890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573749    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-workload-socket\") pod \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") "
Oct 30 01:53:44.573890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573784    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc75808d-fb11-4820-939a-18037b0c2791-istio-token\" (UniqueName: \"kubernetes.io/projected/bc75808d-fb11-4820-939a-18037b0c2791-istio-token\") pod \"bc75808d-fb11-4820-939a-18037b0c2791\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") "
Oct 30 01:53:44.573890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573789    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-credential-socket" pod "89b1105d-0395-4e80-a63b-7e8e0cce276d" (UID: "89b1105d-0395-4e80-a63b-7e8e0cce276d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.573890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573811    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-token\" (UniqueName: \"kubernetes.io/projected/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-token\") pod \"89b1105d-0395-4e80-a63b-7e8e0cce276d\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") "
Oct 30 01:53:44.573890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573842    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-workload-certs\") pod \"89b1105d-0395-4e80-a63b-7e8e0cce276d\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") "
Oct 30 01:53:44.573890 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573894    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e92bdf87-b7d6-4060-9615-70ae77214bc0-kube-api-access-lxkl8\" (UniqueName: \"kubernetes.io/projected/e92bdf87-b7d6-4060-9615-70ae77214bc0-kube-api-access-lxkl8\") pod \"e92bdf87-b7d6-4060-9615-70ae77214bc0\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") "
Oct 30 01:53:44.574962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573936    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-workload-socket" pod "bc75808d-fb11-4820-939a-18037b0c2791" (UID: "bc75808d-fb11-4820-939a-18037b0c2791"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.574962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.573973    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-envoy\") pod \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") "
Oct 30 01:53:44.574962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.574021    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc75808d-fb11-4820-939a-18037b0c2791-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc75808d-fb11-4820-939a-18037b0c2791-istiod-ca-cert\") pod \"bc75808d-fb11-4820-939a-18037b0c2791\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") "
Oct 30 01:53:44.574962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.574045    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-data" pod "9de67abe-4f7c-441f-9e3e-cf34c74068d9" (UID: "9de67abe-4f7c-441f-9e3e-cf34c74068d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.574962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.574106    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-workload-certs\") pod \"bc75808d-fb11-4820-939a-18037b0c2791\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") "
Oct 30 01:53:44.574962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.574166    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-podinfo\") pod \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\" (UID: \"9de67abe-4f7c-441f-9e3e-cf34c74068d9\") "
Oct 30 01:53:44.574962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.574209    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-envoy\") pod \"e92bdf87-b7d6-4060-9615-70ae77214bc0\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") "
Oct 30 01:53:44.574962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.574252    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-istio-envoy\") pod \"bc75808d-fb11-4820-939a-18037b0c2791\" (UID: \"bc75808d-fb11-4820-939a-18037b0c2791\") "
Oct 30 01:53:44.574962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.574255    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-workload-socket" pod "e92bdf87-b7d6-4060-9615-70ae77214bc0" (UID: "e92bdf87-b7d6-4060-9615-70ae77214bc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.574962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.574291    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b1105d-0395-4e80-a63b-7e8e0cce276d-kube-api-access-5pjbq\" (UniqueName: \"kubernetes.io/projected/89b1105d-0395-4e80-a63b-7e8e0cce276d-kube-api-access-5pjbq\") pod \"89b1105d-0395-4e80-a63b-7e8e0cce276d\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") "
Oct 30 01:53:44.574962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.574316    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-data" pod "89b1105d-0395-4e80-a63b-7e8e0cce276d" (UID: "89b1105d-0395-4e80-a63b-7e8e0cce276d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.574962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.574326    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-podinfo\") pod \"e92bdf87-b7d6-4060-9615-70ae77214bc0\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") "
Oct 30 01:53:44.574962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.574373    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b1105d-0395-4e80-a63b-7e8e0cce276d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b1105d-0395-4e80-a63b-7e8e0cce276d-istiod-ca-cert\") pod \"89b1105d-0395-4e80-a63b-7e8e0cce276d\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") "
Oct 30 01:53:44.574962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.574417    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-data\") pod \"e92bdf87-b7d6-4060-9615-70ae77214bc0\" (UID: \"e92bdf87-b7d6-4060-9615-70ae77214bc0\") "
Oct 30 01:53:44.574962 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.574451    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-envoy\") pod \"89b1105d-0395-4e80-a63b-7e8e0cce276d\" (UID: \"89b1105d-0395-4e80-a63b-7e8e0cce276d\") "
Oct 30 01:53:44.575809 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.575156    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.575809 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.575182    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.575809 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.575198    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.575809 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.575214    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.575809 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.575227    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92bdf87-b7d6-4060-9615-70ae77214bc0-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.575809 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.575240    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.575809 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.575253    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.575809 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.575268    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.575809 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.575282    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.575809 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.575296    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.577391 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.574329    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-istio-data" pod "bc75808d-fb11-4820-939a-18037b0c2791" (UID: "bc75808d-fb11-4820-939a-18037b0c2791"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.577391 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.574477    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-workload-certs" pod "9de67abe-4f7c-441f-9e3e-cf34c74068d9" (UID: "9de67abe-4f7c-441f-9e3e-cf34c74068d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.577563 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.574905    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-credential-socket" pod "bc75808d-fb11-4820-939a-18037b0c2791" (UID: "bc75808d-fb11-4820-939a-18037b0c2791"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.577563 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.575110    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-workload-socket" pod "89b1105d-0395-4e80-a63b-7e8e0cce276d" (UID: "89b1105d-0395-4e80-a63b-7e8e0cce276d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.577563 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.575151    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-workload-certs" pod "bc75808d-fb11-4820-939a-18037b0c2791" (UID: "bc75808d-fb11-4820-939a-18037b0c2791"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.577563 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.575408    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-workload-socket" pod "9de67abe-4f7c-441f-9e3e-cf34c74068d9" (UID: "9de67abe-4f7c-441f-9e3e-cf34c74068d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.577563 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.575847    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de67abe-4f7c-441f-9e3e-cf34c74068d9-kube-api-access-7f9vf" pod "9de67abe-4f7c-441f-9e3e-cf34c74068d9" (UID: "9de67abe-4f7c-441f-9e3e-cf34c74068d9"). InnerVolumeSpecName "kube-api-access-7f9vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:44.577563 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.577047    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-data" pod "e92bdf87-b7d6-4060-9615-70ae77214bc0" (UID: "e92bdf87-b7d6-4060-9615-70ae77214bc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.577563 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.577197    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b1105d-0395-4e80-a63b-7e8e0cce276d-istiod-ca-cert" pod "89b1105d-0395-4e80-a63b-7e8e0cce276d" (UID: "89b1105d-0395-4e80-a63b-7e8e0cce276d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:53:44.578511 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.578452    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc75808d-fb11-4820-939a-18037b0c2791-istiod-ca-cert" pod "bc75808d-fb11-4820-939a-18037b0c2791" (UID: "bc75808d-fb11-4820-939a-18037b0c2791"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:53:44.578814 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.578786    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-workload-certs" pod "89b1105d-0395-4e80-a63b-7e8e0cce276d" (UID: "89b1105d-0395-4e80-a63b-7e8e0cce276d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.579612 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.579585    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-token" pod "9de67abe-4f7c-441f-9e3e-cf34c74068d9" (UID: "9de67abe-4f7c-441f-9e3e-cf34c74068d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:44.579612 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.579584    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-envoy" pod "89b1105d-0395-4e80-a63b-7e8e0cce276d" (UID: "89b1105d-0395-4e80-a63b-7e8e0cce276d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.580597 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.580573    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc75808d-fb11-4820-939a-18037b0c2791-kube-api-access-zj9zf" pod "bc75808d-fb11-4820-939a-18037b0c2791" (UID: "bc75808d-fb11-4820-939a-18037b0c2791"). InnerVolumeSpecName "kube-api-access-zj9zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:44.580859 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.580835    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc75808d-fb11-4820-939a-18037b0c2791-istio-podinfo" pod "bc75808d-fb11-4820-939a-18037b0c2791" (UID: "bc75808d-fb11-4820-939a-18037b0c2791"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:53:44.582375 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.582349    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-envoy" pod "e92bdf87-b7d6-4060-9615-70ae77214bc0" (UID: "e92bdf87-b7d6-4060-9615-70ae77214bc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.582469 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.582457    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-istio-envoy" pod "bc75808d-fb11-4820-939a-18037b0c2791" (UID: "bc75808d-fb11-4820-939a-18037b0c2791"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.582533 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.582463    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-podinfo" pod "e92bdf87-b7d6-4060-9615-70ae77214bc0" (UID: "e92bdf87-b7d6-4060-9615-70ae77214bc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:53:44.582533 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.582461    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc75808d-fb11-4820-939a-18037b0c2791-istio-token" pod "bc75808d-fb11-4820-939a-18037b0c2791" (UID: "bc75808d-fb11-4820-939a-18037b0c2791"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:44.582658 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.582548    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b1105d-0395-4e80-a63b-7e8e0cce276d-kube-api-access-5pjbq" pod "89b1105d-0395-4e80-a63b-7e8e0cce276d" (UID: "89b1105d-0395-4e80-a63b-7e8e0cce276d"). InnerVolumeSpecName "kube-api-access-5pjbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:44.582658 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.582586    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-podinfo" pod "9de67abe-4f7c-441f-9e3e-cf34c74068d9" (UID: "9de67abe-4f7c-441f-9e3e-cf34c74068d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:53:44.582658 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.582636    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-podinfo" pod "89b1105d-0395-4e80-a63b-7e8e0cce276d" (UID: "89b1105d-0395-4e80-a63b-7e8e0cce276d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:53:44.582809 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.582658    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92bdf87-b7d6-4060-9615-70ae77214bc0-kube-api-access-lxkl8" pod "e92bdf87-b7d6-4060-9615-70ae77214bc0" (UID: "e92bdf87-b7d6-4060-9615-70ae77214bc0"). InnerVolumeSpecName "kube-api-access-lxkl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:44.582809 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.582679    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-token" pod "e92bdf87-b7d6-4060-9615-70ae77214bc0" (UID: "e92bdf87-b7d6-4060-9615-70ae77214bc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:44.582809 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.582749    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-token" pod "89b1105d-0395-4e80-a63b-7e8e0cce276d" (UID: "89b1105d-0395-4e80-a63b-7e8e0cce276d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:44.584017 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.583996    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-envoy" pod "9de67abe-4f7c-441f-9e3e-cf34c74068d9" (UID: "9de67abe-4f7c-441f-9e3e-cf34c74068d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.676670 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676590    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc75808d-fb11-4820-939a-18037b0c2791-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.676670 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676617    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.676670 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676626    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.676670 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676635    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.676670 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676644    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.676670 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676652    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pjbq\" (UniqueName: \"kubernetes.io/projected/89b1105d-0395-4e80-a63b-7e8e0cce276d-kube-api-access-5pjbq\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.676670 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676660    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.676670 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676669    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b1105d-0395-4e80-a63b-7e8e0cce276d-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.677058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676681    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.677058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676689    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.677058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676696    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f9vf\" (UniqueName: \"kubernetes.io/projected/9de67abe-4f7c-441f-9e3e-cf34c74068d9-kube-api-access-7f9vf\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.677058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676705    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj9zf\" (UniqueName: \"kubernetes.io/projected/bc75808d-fb11-4820-939a-18037b0c2791-kube-api-access-zj9zf\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.677058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676713    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc75808d-fb11-4820-939a-18037b0c2791-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.677058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676720    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92bdf87-b7d6-4060-9615-70ae77214bc0-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.677058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676727    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.677058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676735    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.677058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676743    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.677058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676750    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.677058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676758    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.677058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676768    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc75808d-fb11-4820-939a-18037b0c2791-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.677058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676776    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.677058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676784    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc75808d-fb11-4820-939a-18037b0c2791-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.677058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676793    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b1105d-0395-4e80-a63b-7e8e0cce276d-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.677058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676800    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b1105d-0395-4e80-a63b-7e8e0cce276d-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.677058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676808    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxkl8\" (UniqueName: \"kubernetes.io/projected/e92bdf87-b7d6-4060-9615-70ae77214bc0-kube-api-access-lxkl8\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.677058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.676816    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de67abe-4f7c-441f-9e3e-cf34c74068d9-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.722188 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.722143    4864 generic.go:356] "Generic (PLEG): container finished" podID="e92bdf87-b7d6-4060-9615-70ae77214bc0" containerID="f87e5624c1595c3e1095260cb5fa030036ab26d533f8eaddf333d1a776c477a6" exitCode=0
Oct 30 01:53:44.722387 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.722189    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s" event={"ID":"e92bdf87-b7d6-4060-9615-70ae77214bc0","Type":"ContainerDied","Data":"f87e5624c1595c3e1095260cb5fa030036ab26d533f8eaddf333d1a776c477a6"}
Oct 30 01:53:44.722387 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.722225    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"
Oct 30 01:53:44.722387 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.722247    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s" event={"ID":"e92bdf87-b7d6-4060-9615-70ae77214bc0","Type":"ContainerDied","Data":"d4b7fc3593115050a7701bf19ace7603d52b83e20c412c6b2127b3cc73a11591"}
Oct 30 01:53:44.722387 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.722274    4864 scope.go:122] "RemoveContainer" containerID="f87e5624c1595c3e1095260cb5fa030036ab26d533f8eaddf333d1a776c477a6"
Oct 30 01:53:44.723634 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.723615    4864 generic.go:356] "Generic (PLEG): container finished" podID="9de67abe-4f7c-441f-9e3e-cf34c74068d9" containerID="fa4d3a238a1c681ec10393a53e58be3242307bd7f9be38200f339e40cec41164" exitCode=0
Oct 30 01:53:44.723736 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.723667    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"
Oct 30 01:53:44.723803 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.723672    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld" event={"ID":"9de67abe-4f7c-441f-9e3e-cf34c74068d9","Type":"ContainerDied","Data":"fa4d3a238a1c681ec10393a53e58be3242307bd7f9be38200f339e40cec41164"}
Oct 30 01:53:44.723865 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.723817    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld" event={"ID":"9de67abe-4f7c-441f-9e3e-cf34c74068d9","Type":"ContainerDied","Data":"aeebcc32137e2b837da9ced4300bbb54335067431fc4031bdb52dad68eda4705"}
Oct 30 01:53:44.725004 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.724981    4864 generic.go:356] "Generic (PLEG): container finished" podID="89b1105d-0395-4e80-a63b-7e8e0cce276d" containerID="ca24b5ad03af3279d18cb6f628f6947a3928918cf5028902146eaa2e6c999ce1" exitCode=0
Oct 30 01:53:44.725141 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.725019    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"
Oct 30 01:53:44.725141 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.725120    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t" event={"ID":"89b1105d-0395-4e80-a63b-7e8e0cce276d","Type":"ContainerDied","Data":"ca24b5ad03af3279d18cb6f628f6947a3928918cf5028902146eaa2e6c999ce1"}
Oct 30 01:53:44.725269 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.725160    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t" event={"ID":"89b1105d-0395-4e80-a63b-7e8e0cce276d","Type":"ContainerDied","Data":"11afd25fb8bd86920a28f43aeb5c7d0f094adeb8671e058b8e87c979ee966b08"}
Oct 30 01:53:44.726707 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.726646    4864 generic.go:356] "Generic (PLEG): container finished" podID="bc75808d-fb11-4820-939a-18037b0c2791" containerID="49b43214c8922cf236068aa83328006e70e5f4b5d6c9a6d70a55803393cf0e2f" exitCode=0
Oct 30 01:53:44.726707 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.726683    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx" event={"ID":"bc75808d-fb11-4820-939a-18037b0c2791","Type":"ContainerDied","Data":"49b43214c8922cf236068aa83328006e70e5f4b5d6c9a6d70a55803393cf0e2f"}
Oct 30 01:53:44.726864 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.726731    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx" event={"ID":"bc75808d-fb11-4820-939a-18037b0c2791","Type":"ContainerDied","Data":"506e3770a45f1fa060a26cae8ec1684d0101c981a9cc1ad1c0d51fa556dd142f"}
Oct 30 01:53:44.726864 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.726692    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"
Oct 30 01:53:44.735333 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.735312    4864 scope.go:122] "RemoveContainer" containerID="f87e5624c1595c3e1095260cb5fa030036ab26d533f8eaddf333d1a776c477a6"
Oct 30 01:53:44.735692 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:53:44.735664    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f87e5624c1595c3e1095260cb5fa030036ab26d533f8eaddf333d1a776c477a6\": container with ID starting with f87e5624c1595c3e1095260cb5fa030036ab26d533f8eaddf333d1a776c477a6 not found: ID does not exist" containerID="f87e5624c1595c3e1095260cb5fa030036ab26d533f8eaddf333d1a776c477a6"
Oct 30 01:53:44.735790 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.735703    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f87e5624c1595c3e1095260cb5fa030036ab26d533f8eaddf333d1a776c477a6"} err="failed to get container status \"f87e5624c1595c3e1095260cb5fa030036ab26d533f8eaddf333d1a776c477a6\": rpc error: code = NotFound desc = could not find container \"f87e5624c1595c3e1095260cb5fa030036ab26d533f8eaddf333d1a776c477a6\": container with ID starting with f87e5624c1595c3e1095260cb5fa030036ab26d533f8eaddf333d1a776c477a6 not found: ID does not exist"
Oct 30 01:53:44.735790 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.735725    4864 scope.go:122] "RemoveContainer" containerID="fa4d3a238a1c681ec10393a53e58be3242307bd7f9be38200f339e40cec41164"
Oct 30 01:53:44.746156 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.746141    4864 scope.go:122] "RemoveContainer" containerID="fa4d3a238a1c681ec10393a53e58be3242307bd7f9be38200f339e40cec41164"
Oct 30 01:53:44.746421 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:53:44.746396    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa4d3a238a1c681ec10393a53e58be3242307bd7f9be38200f339e40cec41164\": container with ID starting with fa4d3a238a1c681ec10393a53e58be3242307bd7f9be38200f339e40cec41164 not found: ID does not exist" containerID="fa4d3a238a1c681ec10393a53e58be3242307bd7f9be38200f339e40cec41164"
Oct 30 01:53:44.746467 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.746431    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa4d3a238a1c681ec10393a53e58be3242307bd7f9be38200f339e40cec41164"} err="failed to get container status \"fa4d3a238a1c681ec10393a53e58be3242307bd7f9be38200f339e40cec41164\": rpc error: code = NotFound desc = could not find container \"fa4d3a238a1c681ec10393a53e58be3242307bd7f9be38200f339e40cec41164\": container with ID starting with fa4d3a238a1c681ec10393a53e58be3242307bd7f9be38200f339e40cec41164 not found: ID does not exist"
Oct 30 01:53:44.746467 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.746450    4864 scope.go:122] "RemoveContainer" containerID="ca24b5ad03af3279d18cb6f628f6947a3928918cf5028902146eaa2e6c999ce1"
Oct 30 01:53:44.755652 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.755633    4864 scope.go:122] "RemoveContainer" containerID="ca24b5ad03af3279d18cb6f628f6947a3928918cf5028902146eaa2e6c999ce1"
Oct 30 01:53:44.755895 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:53:44.755874    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca24b5ad03af3279d18cb6f628f6947a3928918cf5028902146eaa2e6c999ce1\": container with ID starting with ca24b5ad03af3279d18cb6f628f6947a3928918cf5028902146eaa2e6c999ce1 not found: ID does not exist" containerID="ca24b5ad03af3279d18cb6f628f6947a3928918cf5028902146eaa2e6c999ce1"
Oct 30 01:53:44.755978 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.755907    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca24b5ad03af3279d18cb6f628f6947a3928918cf5028902146eaa2e6c999ce1"} err="failed to get container status \"ca24b5ad03af3279d18cb6f628f6947a3928918cf5028902146eaa2e6c999ce1\": rpc error: code = NotFound desc = could not find container \"ca24b5ad03af3279d18cb6f628f6947a3928918cf5028902146eaa2e6c999ce1\": container with ID starting with ca24b5ad03af3279d18cb6f628f6947a3928918cf5028902146eaa2e6c999ce1 not found: ID does not exist"
Oct 30 01:53:44.755978 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.755930    4864 scope.go:122] "RemoveContainer" containerID="49b43214c8922cf236068aa83328006e70e5f4b5d6c9a6d70a55803393cf0e2f"
Oct 30 01:53:44.764955 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.764938    4864 scope.go:122] "RemoveContainer" containerID="49b43214c8922cf236068aa83328006e70e5f4b5d6c9a6d70a55803393cf0e2f"
Oct 30 01:53:44.765276 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:53:44.765257    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49b43214c8922cf236068aa83328006e70e5f4b5d6c9a6d70a55803393cf0e2f\": container with ID starting with 49b43214c8922cf236068aa83328006e70e5f4b5d6c9a6d70a55803393cf0e2f not found: ID does not exist" containerID="49b43214c8922cf236068aa83328006e70e5f4b5d6c9a6d70a55803393cf0e2f"
Oct 30 01:53:44.765345 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.765284    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49b43214c8922cf236068aa83328006e70e5f4b5d6c9a6d70a55803393cf0e2f"} err="failed to get container status \"49b43214c8922cf236068aa83328006e70e5f4b5d6c9a6d70a55803393cf0e2f\": rpc error: code = NotFound desc = could not find container \"49b43214c8922cf236068aa83328006e70e5f4b5d6c9a6d70a55803393cf0e2f\": container with ID starting with 49b43214c8922cf236068aa83328006e70e5f4b5d6c9a6d70a55803393cf0e2f not found: ID does not exist"
Oct 30 01:53:44.814252 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.814225    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"]
Oct 30 01:53:44.821563 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.821538    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zct5s"]
Oct 30 01:53:44.853233 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.853208    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"]
Oct 30 01:53:44.859462 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.859437    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6hcmx"]
Oct 30 01:53:44.891893 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.891863    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"]
Oct 30 01:53:44.901823 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.901797    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-89zld"]
Oct 30 01:53:44.938600 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.938533    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"]
Oct 30 01:53:44.946339 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:44.946314    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vwf7t"]
Oct 30 01:53:45.793855 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:45.793820    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b1105d-0395-4e80-a63b-7e8e0cce276d" path="/var/lib/kubelet/pods/89b1105d-0395-4e80-a63b-7e8e0cce276d/volumes"
Oct 30 01:53:45.794485 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:45.794463    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9de67abe-4f7c-441f-9e3e-cf34c74068d9" path="/var/lib/kubelet/pods/9de67abe-4f7c-441f-9e3e-cf34c74068d9/volumes"
Oct 30 01:53:45.794953 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:45.794933    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc75808d-fb11-4820-939a-18037b0c2791" path="/var/lib/kubelet/pods/bc75808d-fb11-4820-939a-18037b0c2791/volumes"
Oct 30 01:53:45.795757 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:45.795736    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e92bdf87-b7d6-4060-9615-70ae77214bc0" path="/var/lib/kubelet/pods/e92bdf87-b7d6-4060-9615-70ae77214bc0/volumes"
Oct 30 01:53:46.462837 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:46.462806    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"]
Oct 30 01:53:46.463056 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:46.463005    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf" podUID="2752235c-68d2-4be4-b1eb-d851d2b5c903" containerName="maas-api" containerID="cri-o://111a6509afe8eed4eb8a2bf83776bbf3bb681e8ccba1ab05196b097e9a90faa2" gracePeriod=30
Oct 30 01:53:46.745845 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:46.745812    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"
Oct 30 01:53:46.747801 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:46.747773    4864 generic.go:356] "Generic (PLEG): container finished" podID="2752235c-68d2-4be4-b1eb-d851d2b5c903" containerID="111a6509afe8eed4eb8a2bf83776bbf3bb681e8ccba1ab05196b097e9a90faa2" exitCode=0
Oct 30 01:53:46.747985 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:46.747969    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf" event={"ID":"2752235c-68d2-4be4-b1eb-d851d2b5c903","Type":"ContainerDied","Data":"111a6509afe8eed4eb8a2bf83776bbf3bb681e8ccba1ab05196b097e9a90faa2"}
Oct 30 01:53:46.748066 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:46.748052    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf" event={"ID":"2752235c-68d2-4be4-b1eb-d851d2b5c903","Type":"ContainerDied","Data":"79fa677f24f978855379a1ea8058abeb5af843e168221f751e99a4e06c6950ce"}
Oct 30 01:53:46.748128 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:46.748101    4864 scope.go:122] "RemoveContainer" containerID="111a6509afe8eed4eb8a2bf83776bbf3bb681e8ccba1ab05196b097e9a90faa2"
Oct 30 01:53:46.748245 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:46.748232    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"
Oct 30 01:53:46.758636 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:46.758606    4864 scope.go:122] "RemoveContainer" containerID="111a6509afe8eed4eb8a2bf83776bbf3bb681e8ccba1ab05196b097e9a90faa2"
Oct 30 01:53:46.758923 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:53:46.758895    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"111a6509afe8eed4eb8a2bf83776bbf3bb681e8ccba1ab05196b097e9a90faa2\": container with ID starting with 111a6509afe8eed4eb8a2bf83776bbf3bb681e8ccba1ab05196b097e9a90faa2 not found: ID does not exist" containerID="111a6509afe8eed4eb8a2bf83776bbf3bb681e8ccba1ab05196b097e9a90faa2"
Oct 30 01:53:46.759032 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:46.758930    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"111a6509afe8eed4eb8a2bf83776bbf3bb681e8ccba1ab05196b097e9a90faa2"} err="failed to get container status \"111a6509afe8eed4eb8a2bf83776bbf3bb681e8ccba1ab05196b097e9a90faa2\": rpc error: code = NotFound desc = could not find container \"111a6509afe8eed4eb8a2bf83776bbf3bb681e8ccba1ab05196b097e9a90faa2\": container with ID starting with 111a6509afe8eed4eb8a2bf83776bbf3bb681e8ccba1ab05196b097e9a90faa2 not found: ID does not exist"
Oct 30 01:53:46.903795 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:46.903762    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2752235c-68d2-4be4-b1eb-d851d2b5c903-kube-api-access-hzhrq\" (UniqueName: \"kubernetes.io/projected/2752235c-68d2-4be4-b1eb-d851d2b5c903-kube-api-access-hzhrq\") pod \"2752235c-68d2-4be4-b1eb-d851d2b5c903\" (UID: \"2752235c-68d2-4be4-b1eb-d851d2b5c903\") "
Oct 30 01:53:46.904325 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:46.903847    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2752235c-68d2-4be4-b1eb-d851d2b5c903-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2752235c-68d2-4be4-b1eb-d851d2b5c903-metrics-certs\") pod \"2752235c-68d2-4be4-b1eb-d851d2b5c903\" (UID: \"2752235c-68d2-4be4-b1eb-d851d2b5c903\") "
Oct 30 01:53:46.904325 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:46.903921    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2752235c-68d2-4be4-b1eb-d851d2b5c903-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2752235c-68d2-4be4-b1eb-d851d2b5c903-openshift-service-ca\") pod \"2752235c-68d2-4be4-b1eb-d851d2b5c903\" (UID: \"2752235c-68d2-4be4-b1eb-d851d2b5c903\") "
Oct 30 01:53:46.904325 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:46.904003    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2752235c-68d2-4be4-b1eb-d851d2b5c903-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2752235c-68d2-4be4-b1eb-d851d2b5c903-maas-api-tls\") pod \"2752235c-68d2-4be4-b1eb-d851d2b5c903\" (UID: \"2752235c-68d2-4be4-b1eb-d851d2b5c903\") "
Oct 30 01:53:46.904325 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:46.904310    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2752235c-68d2-4be4-b1eb-d851d2b5c903-openshift-service-ca" pod "2752235c-68d2-4be4-b1eb-d851d2b5c903" (UID: "2752235c-68d2-4be4-b1eb-d851d2b5c903"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:53:46.904705 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:46.904679    4864 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2752235c-68d2-4be4-b1eb-d851d2b5c903-openshift-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:46.906086 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:46.906048    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2752235c-68d2-4be4-b1eb-d851d2b5c903-maas-api-tls" pod "2752235c-68d2-4be4-b1eb-d851d2b5c903" (UID: "2752235c-68d2-4be4-b1eb-d851d2b5c903"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:53:46.906488 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:46.906469    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2752235c-68d2-4be4-b1eb-d851d2b5c903-metrics-certs" pod "2752235c-68d2-4be4-b1eb-d851d2b5c903" (UID: "2752235c-68d2-4be4-b1eb-d851d2b5c903"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:53:46.906566 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:46.906497    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2752235c-68d2-4be4-b1eb-d851d2b5c903-kube-api-access-hzhrq" pod "2752235c-68d2-4be4-b1eb-d851d2b5c903" (UID: "2752235c-68d2-4be4-b1eb-d851d2b5c903"). InnerVolumeSpecName "kube-api-access-hzhrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:47.005786 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:47.005708    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2752235c-68d2-4be4-b1eb-d851d2b5c903-maas-api-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:47.005786 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:47.005738    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzhrq\" (UniqueName: \"kubernetes.io/projected/2752235c-68d2-4be4-b1eb-d851d2b5c903-kube-api-access-hzhrq\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:47.005786 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:47.005749    4864 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2752235c-68d2-4be4-b1eb-d851d2b5c903-metrics-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:53:47.104596 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:47.104562    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"]
Oct 30 01:53:47.127624 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:47.127591    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a47b9a-7b94478fbb-76lmf"]
Oct 30 01:53:47.795024 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:53:47.794989    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2752235c-68d2-4be4-b1eb-d851d2b5c903" path="/var/lib/kubelet/pods/2752235c-68d2-4be4-b1eb-d851d2b5c903/volumes"
Oct 30 01:53:49.042810 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:53:49.042772    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:54:11.835618 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:11.835538    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"]
Oct 30 01:54:11.845454 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:11.845425    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"]
Oct 30 01:54:17.550422 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:17.550387    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69"]
Oct 30 01:54:18.951156 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:18.951120    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"]
Oct 30 01:54:18.951620 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:18.951340    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f" podUID="9f728c20-eae2-4fd4-aa40-4094a75f5f16" containerName="maas-api" containerID="cri-o://5c123d2e2c324915d0334e534e27bbc6af5548c3a8b925070d85e68635cd8240" gracePeriod=30
Oct 30 01:54:19.042858 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:54:19.042823    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:54:19.252672 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:19.252643    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"
Oct 30 01:54:19.392427 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:19.392390    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f728c20-eae2-4fd4-aa40-4094a75f5f16-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f728c20-eae2-4fd4-aa40-4094a75f5f16-openshift-service-ca\") pod \"9f728c20-eae2-4fd4-aa40-4094a75f5f16\" (UID: \"9f728c20-eae2-4fd4-aa40-4094a75f5f16\") "
Oct 30 01:54:19.392599 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:19.392512    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f728c20-eae2-4fd4-aa40-4094a75f5f16-kube-api-access-khmms\" (UniqueName: \"kubernetes.io/projected/9f728c20-eae2-4fd4-aa40-4094a75f5f16-kube-api-access-khmms\") pod \"9f728c20-eae2-4fd4-aa40-4094a75f5f16\" (UID: \"9f728c20-eae2-4fd4-aa40-4094a75f5f16\") "
Oct 30 01:54:19.392643 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:19.392606    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f728c20-eae2-4fd4-aa40-4094a75f5f16-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f728c20-eae2-4fd4-aa40-4094a75f5f16-maas-api-tls\") pod \"9f728c20-eae2-4fd4-aa40-4094a75f5f16\" (UID: \"9f728c20-eae2-4fd4-aa40-4094a75f5f16\") "
Oct 30 01:54:19.392643 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:19.392629    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f728c20-eae2-4fd4-aa40-4094a75f5f16-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f728c20-eae2-4fd4-aa40-4094a75f5f16-metrics-certs\") pod \"9f728c20-eae2-4fd4-aa40-4094a75f5f16\" (UID: \"9f728c20-eae2-4fd4-aa40-4094a75f5f16\") "
Oct 30 01:54:19.392758 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:19.392735    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f728c20-eae2-4fd4-aa40-4094a75f5f16-openshift-service-ca" pod "9f728c20-eae2-4fd4-aa40-4094a75f5f16" (UID: "9f728c20-eae2-4fd4-aa40-4094a75f5f16"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:54:19.393058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:19.393042    4864 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f728c20-eae2-4fd4-aa40-4094a75f5f16-openshift-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:19.394733 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:19.394700    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f728c20-eae2-4fd4-aa40-4094a75f5f16-maas-api-tls" pod "9f728c20-eae2-4fd4-aa40-4094a75f5f16" (UID: "9f728c20-eae2-4fd4-aa40-4094a75f5f16"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:54:19.394851 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:19.394745    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f728c20-eae2-4fd4-aa40-4094a75f5f16-metrics-certs" pod "9f728c20-eae2-4fd4-aa40-4094a75f5f16" (UID: "9f728c20-eae2-4fd4-aa40-4094a75f5f16"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:54:19.394851 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:19.394761    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f728c20-eae2-4fd4-aa40-4094a75f5f16-kube-api-access-khmms" pod "9f728c20-eae2-4fd4-aa40-4094a75f5f16" (UID: "9f728c20-eae2-4fd4-aa40-4094a75f5f16"). InnerVolumeSpecName "kube-api-access-khmms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:54:19.494690 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:19.494603    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khmms\" (UniqueName: \"kubernetes.io/projected/9f728c20-eae2-4fd4-aa40-4094a75f5f16-kube-api-access-khmms\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:19.494690 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:19.494643    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f728c20-eae2-4fd4-aa40-4094a75f5f16-maas-api-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:19.494690 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:19.494658    4864 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f728c20-eae2-4fd4-aa40-4094a75f5f16-metrics-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:19.957993 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:19.957954    4864 generic.go:356] "Generic (PLEG): container finished" podID="9f728c20-eae2-4fd4-aa40-4094a75f5f16" containerID="5c123d2e2c324915d0334e534e27bbc6af5548c3a8b925070d85e68635cd8240" exitCode=0
Oct 30 01:54:19.958445 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:19.957998    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"
Oct 30 01:54:19.958445 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:19.958040    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f" event={"ID":"9f728c20-eae2-4fd4-aa40-4094a75f5f16","Type":"ContainerDied","Data":"5c123d2e2c324915d0334e534e27bbc6af5548c3a8b925070d85e68635cd8240"}
Oct 30 01:54:19.958445 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:19.958126    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f" event={"ID":"9f728c20-eae2-4fd4-aa40-4094a75f5f16","Type":"ContainerDied","Data":"3dc32146bd25d14d0fa50285e3d64994aa8c70aef448ce0ebdfdfaa44cc68bed"}
Oct 30 01:54:19.958445 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:19.958156    4864 scope.go:122] "RemoveContainer" containerID="5c123d2e2c324915d0334e534e27bbc6af5548c3a8b925070d85e68635cd8240"
Oct 30 01:54:19.968361 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:19.968342    4864 scope.go:122] "RemoveContainer" containerID="5c123d2e2c324915d0334e534e27bbc6af5548c3a8b925070d85e68635cd8240"
Oct 30 01:54:19.968659 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:54:19.968639    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c123d2e2c324915d0334e534e27bbc6af5548c3a8b925070d85e68635cd8240\": container with ID starting with 5c123d2e2c324915d0334e534e27bbc6af5548c3a8b925070d85e68635cd8240 not found: ID does not exist" containerID="5c123d2e2c324915d0334e534e27bbc6af5548c3a8b925070d85e68635cd8240"
Oct 30 01:54:19.968752 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:19.968667    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c123d2e2c324915d0334e534e27bbc6af5548c3a8b925070d85e68635cd8240"} err="failed to get container status \"5c123d2e2c324915d0334e534e27bbc6af5548c3a8b925070d85e68635cd8240\": rpc error: code = NotFound desc = could not find container \"5c123d2e2c324915d0334e534e27bbc6af5548c3a8b925070d85e68635cd8240\": container with ID starting with 5c123d2e2c324915d0334e534e27bbc6af5548c3a8b925070d85e68635cd8240 not found: ID does not exist"
Oct 30 01:54:20.004997 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:20.004966    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"]
Oct 30 01:54:20.014807 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:20.014783    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0e1a98-86bb7b4985-s9k4f"]
Oct 30 01:54:21.793189 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:21.793153    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f728c20-eae2-4fd4-aa40-4094a75f5f16" path="/var/lib/kubelet/pods/9f728c20-eae2-4fd4-aa40-4094a75f5f16/volumes"
Oct 30 01:54:22.550931 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:22.550878    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69" podUID="9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b" containerName="payload-pre-processing" containerID="cri-o://8916621cb9bdd2444ab199e5f6ae7931aa8422cd534e82929bc6bd8a9ba7f794" gracePeriod=25
Oct 30 01:54:26.856154 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:26.856110    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj" podUID="63a8525d-9951-4f19-b124-62e4fa91c6b9" containerName="main" containerID="cri-o://6502f4b99f6280f96497ac9b1f232328a77327da71dca13f37c8b652b50ec961" gracePeriod=45
Oct 30 01:54:26.865568 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:26.865528    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj" podUID="8b267884-baae-4ad3-a414-82467f7188b3" containerName="main" containerID="cri-o://10ba0f967830bb399f6d3ea3615ba77b0b60b2e8c4fb47be3154669f50da7a02" gracePeriod=45
Oct 30 01:54:26.917631 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:26.917595    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj" podUID="63a8525d-9951-4f19-b124-62e4fa91c6b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 30 01:54:27.015109 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.013151    4864 generic.go:356] "Generic (PLEG): container finished" podID="8b267884-baae-4ad3-a414-82467f7188b3" containerID="10ba0f967830bb399f6d3ea3615ba77b0b60b2e8c4fb47be3154669f50da7a02" exitCode=0
Oct 30 01:54:27.015109 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.013254    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj" event={"ID":"8b267884-baae-4ad3-a414-82467f7188b3","Type":"ContainerDied","Data":"10ba0f967830bb399f6d3ea3615ba77b0b60b2e8c4fb47be3154669f50da7a02"}
Oct 30 01:54:27.015109 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.015034    4864 generic.go:356] "Generic (PLEG): container finished" podID="63a8525d-9951-4f19-b124-62e4fa91c6b9" containerID="6502f4b99f6280f96497ac9b1f232328a77327da71dca13f37c8b652b50ec961" exitCode=0
Oct 30 01:54:27.015372 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.015194    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj" event={"ID":"63a8525d-9951-4f19-b124-62e4fa91c6b9","Type":"ContainerDied","Data":"6502f4b99f6280f96497ac9b1f232328a77327da71dca13f37c8b652b50ec961"}
Oct 30 01:54:27.158813 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.158787    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 01:54:27.160808 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.160791    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 01:54:27.189721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.188385    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b267884-baae-4ad3-a414-82467f7188b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/8b267884-baae-4ad3-a414-82467f7188b3-tls-certs\") pod \"8b267884-baae-4ad3-a414-82467f7188b3\" (UID: \"8b267884-baae-4ad3-a414-82467f7188b3\") "
Oct 30 01:54:27.189721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.188449    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63a8525d-9951-4f19-b124-62e4fa91c6b9-kube-api-access-j4c9c\" (UniqueName: \"kubernetes.io/projected/63a8525d-9951-4f19-b124-62e4fa91c6b9-kube-api-access-j4c9c\") pod \"63a8525d-9951-4f19-b124-62e4fa91c6b9\" (UID: \"63a8525d-9951-4f19-b124-62e4fa91c6b9\") "
Oct 30 01:54:27.189721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.188482    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-tmp-dir\") pod \"63a8525d-9951-4f19-b124-62e4fa91c6b9\" (UID: \"63a8525d-9951-4f19-b124-62e4fa91c6b9\") "
Oct 30 01:54:27.189721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.188535    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-model-cache\") pod \"63a8525d-9951-4f19-b124-62e4fa91c6b9\" (UID: \"63a8525d-9951-4f19-b124-62e4fa91c6b9\") "
Oct 30 01:54:27.189721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.188595    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-dshm\") pod \"8b267884-baae-4ad3-a414-82467f7188b3\" (UID: \"8b267884-baae-4ad3-a414-82467f7188b3\") "
Oct 30 01:54:27.189721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.188689    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-dshm\") pod \"63a8525d-9951-4f19-b124-62e4fa91c6b9\" (UID: \"63a8525d-9951-4f19-b124-62e4fa91c6b9\") "
Oct 30 01:54:27.189721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.188724    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b267884-baae-4ad3-a414-82467f7188b3-kube-api-access-z8pxk\" (UniqueName: \"kubernetes.io/projected/8b267884-baae-4ad3-a414-82467f7188b3-kube-api-access-z8pxk\") pod \"8b267884-baae-4ad3-a414-82467f7188b3\" (UID: \"8b267884-baae-4ad3-a414-82467f7188b3\") "
Oct 30 01:54:27.189721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.188747    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-home\" (UniqueName: \"kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-home\") pod \"8b267884-baae-4ad3-a414-82467f7188b3\" (UID: \"8b267884-baae-4ad3-a414-82467f7188b3\") "
Oct 30 01:54:27.189721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.188785    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63a8525d-9951-4f19-b124-62e4fa91c6b9-tls-certs\" (UniqueName: \"kubernetes.io/secret/63a8525d-9951-4f19-b124-62e4fa91c6b9-tls-certs\") pod \"63a8525d-9951-4f19-b124-62e4fa91c6b9\" (UID: \"63a8525d-9951-4f19-b124-62e4fa91c6b9\") "
Oct 30 01:54:27.189721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.188821    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-model-cache\") pod \"8b267884-baae-4ad3-a414-82467f7188b3\" (UID: \"8b267884-baae-4ad3-a414-82467f7188b3\") "
Oct 30 01:54:27.189721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.188872    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-home\" (UniqueName: \"kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-home\") pod \"63a8525d-9951-4f19-b124-62e4fa91c6b9\" (UID: \"63a8525d-9951-4f19-b124-62e4fa91c6b9\") "
Oct 30 01:54:27.189721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.188961    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-tmp-dir\") pod \"8b267884-baae-4ad3-a414-82467f7188b3\" (UID: \"8b267884-baae-4ad3-a414-82467f7188b3\") "
Oct 30 01:54:27.189721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.189409    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-tmp-dir" pod "63a8525d-9951-4f19-b124-62e4fa91c6b9" (UID: "63a8525d-9951-4f19-b124-62e4fa91c6b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:54:27.189721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.189601    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-home" pod "8b267884-baae-4ad3-a414-82467f7188b3" (UID: "8b267884-baae-4ad3-a414-82467f7188b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:54:27.190711 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.190577    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-tmp-dir" pod "8b267884-baae-4ad3-a414-82467f7188b3" (UID: "8b267884-baae-4ad3-a414-82467f7188b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:54:27.190906 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.190880    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-model-cache" pod "63a8525d-9951-4f19-b124-62e4fa91c6b9" (UID: "63a8525d-9951-4f19-b124-62e4fa91c6b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:54:27.191003 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.190949    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-model-cache" pod "8b267884-baae-4ad3-a414-82467f7188b3" (UID: "8b267884-baae-4ad3-a414-82467f7188b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:54:27.191003 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.190997    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-home" pod "63a8525d-9951-4f19-b124-62e4fa91c6b9" (UID: "63a8525d-9951-4f19-b124-62e4fa91c6b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:54:27.193170 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.193144    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b267884-baae-4ad3-a414-82467f7188b3-kube-api-access-z8pxk" pod "8b267884-baae-4ad3-a414-82467f7188b3" (UID: "8b267884-baae-4ad3-a414-82467f7188b3"). InnerVolumeSpecName "kube-api-access-z8pxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:54:27.193304 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.193173    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b267884-baae-4ad3-a414-82467f7188b3-tls-certs" pod "8b267884-baae-4ad3-a414-82467f7188b3" (UID: "8b267884-baae-4ad3-a414-82467f7188b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:54:27.193414 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.193390    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-dshm" pod "8b267884-baae-4ad3-a414-82467f7188b3" (UID: "8b267884-baae-4ad3-a414-82467f7188b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:54:27.193484 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.193437    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-dshm" pod "63a8525d-9951-4f19-b124-62e4fa91c6b9" (UID: "63a8525d-9951-4f19-b124-62e4fa91c6b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:54:27.193953 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.193930    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63a8525d-9951-4f19-b124-62e4fa91c6b9-tls-certs" pod "63a8525d-9951-4f19-b124-62e4fa91c6b9" (UID: "63a8525d-9951-4f19-b124-62e4fa91c6b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:54:27.194884 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.194837    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a8525d-9951-4f19-b124-62e4fa91c6b9-kube-api-access-j4c9c" pod "63a8525d-9951-4f19-b124-62e4fa91c6b9" (UID: "63a8525d-9951-4f19-b124-62e4fa91c6b9"). InnerVolumeSpecName "kube-api-access-j4c9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:54:27.290352 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.290315    4864 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-dshm\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:27.290352 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.290343    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8pxk\" (UniqueName: \"kubernetes.io/projected/8b267884-baae-4ad3-a414-82467f7188b3-kube-api-access-z8pxk\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:27.290352 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.290354    4864 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-home\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:27.290352 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.290363    4864 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63a8525d-9951-4f19-b124-62e4fa91c6b9-tls-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:27.290728 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.290371    4864 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-model-cache\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:27.290728 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.290379    4864 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-home\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:27.290728 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.290386    4864 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-tmp-dir\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:27.290728 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.290393    4864 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b267884-baae-4ad3-a414-82467f7188b3-tls-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:27.290728 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.290400    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4c9c\" (UniqueName: \"kubernetes.io/projected/63a8525d-9951-4f19-b124-62e4fa91c6b9-kube-api-access-j4c9c\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:27.290728 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.290408    4864 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-tmp-dir\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:27.290728 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.290415    4864 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63a8525d-9951-4f19-b124-62e4fa91c6b9-model-cache\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:27.290728 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.290422    4864 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b267884-baae-4ad3-a414-82467f7188b3-dshm\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:27.437851 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.437749    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"]
Oct 30 01:54:27.438031 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:27.437970    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8" podUID="a11b8ed4-c24c-4919-959f-86cf8ecbacb2" containerName="istio-proxy" containerID="cri-o://1a40f0ec4a538e39c27aa6141ed7e275d190a56bed567a5486f160814619efeb" gracePeriod=30
Oct 30 01:54:28.022579 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:28.022550    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"
Oct 30 01:54:28.022579 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:28.022564    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj" event={"ID":"8b267884-baae-4ad3-a414-82467f7188b3","Type":"ContainerDied","Data":"d74099911399f014ffa618601db63083ad37bd8428ee134bedc5fa771061d727"}
Oct 30 01:54:28.023154 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:28.022622    4864 scope.go:122] "RemoveContainer" containerID="10ba0f967830bb399f6d3ea3615ba77b0b60b2e8c4fb47be3154669f50da7a02"
Oct 30 01:54:28.024717 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:28.024635    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj" event={"ID":"63a8525d-9951-4f19-b124-62e4fa91c6b9","Type":"ContainerDied","Data":"8da544390b0e5b084c3c9055600b68195c46c983d6c7f94f79dc47a47a6f0bb5"}
Oct 30 01:54:28.024717 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:28.024675    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"
Oct 30 01:54:28.035385 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:28.035361    4864 scope.go:122] "RemoveContainer" containerID="6502f4b99f6280f96497ac9b1f232328a77327da71dca13f37c8b652b50ec961"
Oct 30 01:54:28.069027 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:28.068962    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"]
Oct 30 01:54:28.078896 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:28.078871    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-39b241/premium-w3-39b241-kserve-54df87b4c8-5kjxj"]
Oct 30 01:54:28.109027 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:28.108989    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"]
Oct 30 01:54:28.111491 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:28.111465    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-39b241/distinct-w3-39b241-kserve-7bc8898fd9-s5qsj"]
Oct 30 01:54:29.393961 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:29.393929    4864 scope.go:122] "RemoveContainer" containerID="79e76a1a8f18ff1b40faa2390aceb61c12ff600ab53bbac4a3ccaa654239f6cf"
Oct 30 01:54:29.794104 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:29.794040    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63a8525d-9951-4f19-b124-62e4fa91c6b9" path="/var/lib/kubelet/pods/63a8525d-9951-4f19-b124-62e4fa91c6b9/volumes"
Oct 30 01:54:29.794575 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:29.794558    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b267884-baae-4ad3-a414-82467f7188b3" path="/var/lib/kubelet/pods/8b267884-baae-4ad3-a414-82467f7188b3/volumes"
Oct 30 01:54:31.278224 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:31.278185    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69" podUID="9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:54:32.705291 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.705231    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:54:32.756047 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.756019    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-data\") pod \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") "
Oct 30 01:54:32.756217 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.756176    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-envoy\") pod \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") "
Oct 30 01:54:32.756217 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.756205    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-kube-api-access-644m5\" (UniqueName: \"kubernetes.io/projected/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-kube-api-access-644m5\") pod \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") "
Oct 30 01:54:32.756342 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.756224    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-workload-socket\") pod \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") "
Oct 30 01:54:32.756342 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.756261    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-workload-certs\") pod \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") "
Oct 30 01:54:32.756342 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.756280    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-data" pod "a11b8ed4-c24c-4919-959f-86cf8ecbacb2" (UID: "a11b8ed4-c24c-4919-959f-86cf8ecbacb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:54:32.756342 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.756292    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istiod-ca-cert\") pod \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") "
Oct 30 01:54:32.756571 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.756379    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-credential-socket\") pod \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") "
Oct 30 01:54:32.756571 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.756436    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-podinfo\") pod \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") "
Oct 30 01:54:32.756571 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.756488    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-token\" (UniqueName: \"kubernetes.io/projected/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-token\") pod \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\" (UID: \"a11b8ed4-c24c-4919-959f-86cf8ecbacb2\") "
Oct 30 01:54:32.756743 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.756599    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-workload-socket" pod "a11b8ed4-c24c-4919-959f-86cf8ecbacb2" (UID: "a11b8ed4-c24c-4919-959f-86cf8ecbacb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:54:32.756743 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.756615    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-workload-certs" pod "a11b8ed4-c24c-4919-959f-86cf8ecbacb2" (UID: "a11b8ed4-c24c-4919-959f-86cf8ecbacb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:54:32.756743 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.756670    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-credential-socket" pod "a11b8ed4-c24c-4919-959f-86cf8ecbacb2" (UID: "a11b8ed4-c24c-4919-959f-86cf8ecbacb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:54:32.756743 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.756691    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istiod-ca-cert" pod "a11b8ed4-c24c-4919-959f-86cf8ecbacb2" (UID: "a11b8ed4-c24c-4919-959f-86cf8ecbacb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:54:32.757384 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.757026    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:32.757384 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.757054    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:32.757384 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.757068    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:32.757384 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.757095    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:32.757384 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.757109    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:32.759122 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.759050    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-envoy" pod "a11b8ed4-c24c-4919-959f-86cf8ecbacb2" (UID: "a11b8ed4-c24c-4919-959f-86cf8ecbacb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:54:32.759238 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.759132    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-podinfo" pod "a11b8ed4-c24c-4919-959f-86cf8ecbacb2" (UID: "a11b8ed4-c24c-4919-959f-86cf8ecbacb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:54:32.759510 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.759485    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-kube-api-access-644m5" pod "a11b8ed4-c24c-4919-959f-86cf8ecbacb2" (UID: "a11b8ed4-c24c-4919-959f-86cf8ecbacb2"). InnerVolumeSpecName "kube-api-access-644m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:54:32.761217 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.761193    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-token" pod "a11b8ed4-c24c-4919-959f-86cf8ecbacb2" (UID: "a11b8ed4-c24c-4919-959f-86cf8ecbacb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:54:32.858059 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.858021    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:32.858059 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.858054    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-644m5\" (UniqueName: \"kubernetes.io/projected/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-kube-api-access-644m5\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:32.858059 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.858063    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:32.858348 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:32.858100    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11b8ed4-c24c-4919-959f-86cf8ecbacb2-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:33.056882 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:33.056798    4864 generic.go:356] "Generic (PLEG): container finished" podID="a11b8ed4-c24c-4919-959f-86cf8ecbacb2" containerID="1a40f0ec4a538e39c27aa6141ed7e275d190a56bed567a5486f160814619efeb" exitCode=0
Oct 30 01:54:33.056882 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:33.056838    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8" event={"ID":"a11b8ed4-c24c-4919-959f-86cf8ecbacb2","Type":"ContainerDied","Data":"1a40f0ec4a538e39c27aa6141ed7e275d190a56bed567a5486f160814619efeb"}
Oct 30 01:54:33.056882 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:33.056881    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8" event={"ID":"a11b8ed4-c24c-4919-959f-86cf8ecbacb2","Type":"ContainerDied","Data":"f1105dedad5826040d7fbeb6cdba266762456107256e204279511121ec7b78a7"}
Oct 30 01:54:33.057151 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:33.056902    4864 scope.go:122] "RemoveContainer" containerID="1a40f0ec4a538e39c27aa6141ed7e275d190a56bed567a5486f160814619efeb"
Oct 30 01:54:33.057151 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:33.056840    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"
Oct 30 01:54:33.068047 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:33.068029    4864 scope.go:122] "RemoveContainer" containerID="1a40f0ec4a538e39c27aa6141ed7e275d190a56bed567a5486f160814619efeb"
Oct 30 01:54:33.068360 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:54:33.068340    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a40f0ec4a538e39c27aa6141ed7e275d190a56bed567a5486f160814619efeb\": container with ID starting with 1a40f0ec4a538e39c27aa6141ed7e275d190a56bed567a5486f160814619efeb not found: ID does not exist" containerID="1a40f0ec4a538e39c27aa6141ed7e275d190a56bed567a5486f160814619efeb"
Oct 30 01:54:33.068422 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:33.068370    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a40f0ec4a538e39c27aa6141ed7e275d190a56bed567a5486f160814619efeb"} err="failed to get container status \"1a40f0ec4a538e39c27aa6141ed7e275d190a56bed567a5486f160814619efeb\": rpc error: code = NotFound desc = could not find container \"1a40f0ec4a538e39c27aa6141ed7e275d190a56bed567a5486f160814619efeb\": container with ID starting with 1a40f0ec4a538e39c27aa6141ed7e275d190a56bed567a5486f160814619efeb not found: ID does not exist"
Oct 30 01:54:33.110986 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:33.110955    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"]
Oct 30 01:54:33.113670 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:33.113644    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0e1a98-openshift-default-7f9b786d5c-rfsr8"]
Oct 30 01:54:33.423730 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:33.423639    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"]
Oct 30 01:54:33.423939 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:33.423908    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh" podUID="778281fb-fe1b-4f27-bd43-62920826a0f9" containerName="istio-proxy" containerID="cri-o://64463487d47622c7848f48f30c8453b651583aed738acaca624ad5c79f94899d" gracePeriod=30
Oct 30 01:54:33.792820 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:33.792783    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a11b8ed4-c24c-4919-959f-86cf8ecbacb2" path="/var/lib/kubelet/pods/a11b8ed4-c24c-4919-959f-86cf8ecbacb2/volumes"
Oct 30 01:54:38.716743 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.716715    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:54:38.824177 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.824103    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-podinfo\") pod \"778281fb-fe1b-4f27-bd43-62920826a0f9\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") "
Oct 30 01:54:38.824320 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.824208    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-data\") pod \"778281fb-fe1b-4f27-bd43-62920826a0f9\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") "
Oct 30 01:54:38.824320 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.824269    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-envoy\") pod \"778281fb-fe1b-4f27-bd43-62920826a0f9\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") "
Oct 30 01:54:38.824320 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.824303    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-credential-socket\") pod \"778281fb-fe1b-4f27-bd43-62920826a0f9\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") "
Oct 30 01:54:38.824429 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.824333    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/778281fb-fe1b-4f27-bd43-62920826a0f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778281fb-fe1b-4f27-bd43-62920826a0f9-istiod-ca-cert\") pod \"778281fb-fe1b-4f27-bd43-62920826a0f9\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") "
Oct 30 01:54:38.824429 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.824377    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/778281fb-fe1b-4f27-bd43-62920826a0f9-kube-api-access-g69kx\" (UniqueName: \"kubernetes.io/projected/778281fb-fe1b-4f27-bd43-62920826a0f9-kube-api-access-g69kx\") pod \"778281fb-fe1b-4f27-bd43-62920826a0f9\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") "
Oct 30 01:54:38.824429 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.824415    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-token\" (UniqueName: \"kubernetes.io/projected/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-token\") pod \"778281fb-fe1b-4f27-bd43-62920826a0f9\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") "
Oct 30 01:54:38.824566 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.824522    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-data" pod "778281fb-fe1b-4f27-bd43-62920826a0f9" (UID: "778281fb-fe1b-4f27-bd43-62920826a0f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:54:38.824632 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.824568    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-credential-socket" pod "778281fb-fe1b-4f27-bd43-62920826a0f9" (UID: "778281fb-fe1b-4f27-bd43-62920826a0f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:54:38.824632 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.824577    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-workload-certs\") pod \"778281fb-fe1b-4f27-bd43-62920826a0f9\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") "
Oct 30 01:54:38.824737 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.824661    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-workload-socket\") pod \"778281fb-fe1b-4f27-bd43-62920826a0f9\" (UID: \"778281fb-fe1b-4f27-bd43-62920826a0f9\") "
Oct 30 01:54:38.824797 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.824772    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/778281fb-fe1b-4f27-bd43-62920826a0f9-istiod-ca-cert" pod "778281fb-fe1b-4f27-bd43-62920826a0f9" (UID: "778281fb-fe1b-4f27-bd43-62920826a0f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:54:38.824911 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.824884    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-workload-certs" pod "778281fb-fe1b-4f27-bd43-62920826a0f9" (UID: "778281fb-fe1b-4f27-bd43-62920826a0f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:54:38.825138 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.825114    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-workload-socket" pod "778281fb-fe1b-4f27-bd43-62920826a0f9" (UID: "778281fb-fe1b-4f27-bd43-62920826a0f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:54:38.825318 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.825298    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:38.825393 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.825324    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:38.825393 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.825341    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:38.825393 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.825356    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:38.825393 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.825371    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778281fb-fe1b-4f27-bd43-62920826a0f9-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:38.826299 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.826277    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-podinfo" pod "778281fb-fe1b-4f27-bd43-62920826a0f9" (UID: "778281fb-fe1b-4f27-bd43-62920826a0f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:54:38.826499 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.826483    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-envoy" pod "778281fb-fe1b-4f27-bd43-62920826a0f9" (UID: "778281fb-fe1b-4f27-bd43-62920826a0f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:54:38.826638 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.826623    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-token" pod "778281fb-fe1b-4f27-bd43-62920826a0f9" (UID: "778281fb-fe1b-4f27-bd43-62920826a0f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:54:38.826709 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.826695    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778281fb-fe1b-4f27-bd43-62920826a0f9-kube-api-access-g69kx" pod "778281fb-fe1b-4f27-bd43-62920826a0f9" (UID: "778281fb-fe1b-4f27-bd43-62920826a0f9"). InnerVolumeSpecName "kube-api-access-g69kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:54:38.926461 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.926425    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g69kx\" (UniqueName: \"kubernetes.io/projected/778281fb-fe1b-4f27-bd43-62920826a0f9-kube-api-access-g69kx\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:38.926461 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.926454    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:38.926461 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.926464    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:38.926461 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:38.926473    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778281fb-fe1b-4f27-bd43-62920826a0f9-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:39.095364 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:39.095279    4864 generic.go:356] "Generic (PLEG): container finished" podID="778281fb-fe1b-4f27-bd43-62920826a0f9" containerID="64463487d47622c7848f48f30c8453b651583aed738acaca624ad5c79f94899d" exitCode=0
Oct 30 01:54:39.095364 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:39.095323    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh" event={"ID":"778281fb-fe1b-4f27-bd43-62920826a0f9","Type":"ContainerDied","Data":"64463487d47622c7848f48f30c8453b651583aed738acaca624ad5c79f94899d"}
Oct 30 01:54:39.095537 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:39.095384    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh" event={"ID":"778281fb-fe1b-4f27-bd43-62920826a0f9","Type":"ContainerDied","Data":"b1ee66ceb752b541bec097dcf52674fbbb7f14d0defe0bd0ffb1cd2980e6e171"}
Oct 30 01:54:39.095537 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:39.095409    4864 scope.go:122] "RemoveContainer" containerID="64463487d47622c7848f48f30c8453b651583aed738acaca624ad5c79f94899d"
Oct 30 01:54:39.095537 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:39.095332    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"
Oct 30 01:54:39.110360 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:39.110338    4864 scope.go:122] "RemoveContainer" containerID="13165003d18702cd7e990b59a64ef4e23adc46333f81f36c2800deabe1c583df"
Oct 30 01:54:39.137350 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:39.137326    4864 scope.go:122] "RemoveContainer" containerID="64463487d47622c7848f48f30c8453b651583aed738acaca624ad5c79f94899d"
Oct 30 01:54:39.137728 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:54:39.137702    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64463487d47622c7848f48f30c8453b651583aed738acaca624ad5c79f94899d\": container with ID starting with 64463487d47622c7848f48f30c8453b651583aed738acaca624ad5c79f94899d not found: ID does not exist" containerID="64463487d47622c7848f48f30c8453b651583aed738acaca624ad5c79f94899d"
Oct 30 01:54:39.137835 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:39.137739    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64463487d47622c7848f48f30c8453b651583aed738acaca624ad5c79f94899d"} err="failed to get container status \"64463487d47622c7848f48f30c8453b651583aed738acaca624ad5c79f94899d\": rpc error: code = NotFound desc = could not find container \"64463487d47622c7848f48f30c8453b651583aed738acaca624ad5c79f94899d\": container with ID starting with 64463487d47622c7848f48f30c8453b651583aed738acaca624ad5c79f94899d not found: ID does not exist"
Oct 30 01:54:39.137835 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:39.137763    4864 scope.go:122] "RemoveContainer" containerID="13165003d18702cd7e990b59a64ef4e23adc46333f81f36c2800deabe1c583df"
Oct 30 01:54:39.138141 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:54:39.138110    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13165003d18702cd7e990b59a64ef4e23adc46333f81f36c2800deabe1c583df\": container with ID starting with 13165003d18702cd7e990b59a64ef4e23adc46333f81f36c2800deabe1c583df not found: ID does not exist" containerID="13165003d18702cd7e990b59a64ef4e23adc46333f81f36c2800deabe1c583df"
Oct 30 01:54:39.138254 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:39.138147    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13165003d18702cd7e990b59a64ef4e23adc46333f81f36c2800deabe1c583df"} err="failed to get container status \"13165003d18702cd7e990b59a64ef4e23adc46333f81f36c2800deabe1c583df\": rpc error: code = NotFound desc = could not find container \"13165003d18702cd7e990b59a64ef4e23adc46333f81f36c2800deabe1c583df\": container with ID starting with 13165003d18702cd7e990b59a64ef4e23adc46333f81f36c2800deabe1c583df not found: ID does not exist"
Oct 30 01:54:39.166531 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:39.166492    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"]
Oct 30 01:54:39.176273 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:39.176239    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-ae1820-openshift-default-c5b57f649-dqfqh"]
Oct 30 01:54:39.793786 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:39.793752    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="778281fb-fe1b-4f27-bd43-62920826a0f9" path="/var/lib/kubelet/pods/778281fb-fe1b-4f27-bd43-62920826a0f9/volumes"
Oct 30 01:54:41.278205 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:41.278162    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69" podUID="9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:54:42.808898 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:42.808841    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69"
Oct 30 01:54:42.872171 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:42.872135    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b-kube-api-access-99nxn\" (UniqueName: \"kubernetes.io/projected/9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b-kube-api-access-99nxn\") pod \"9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b\" (UID: \"9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b\") "
Oct 30 01:54:42.872345 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:42.872194    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b-plugins-config-volume\") pod \"9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b\" (UID: \"9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b\") "
Oct 30 01:54:42.872676 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:42.872641    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b-plugins-config-volume" pod "9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b" (UID: "9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:54:42.872953 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:42.872923    4864 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b-plugins-config-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:42.874481 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:42.874446    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b-kube-api-access-99nxn" pod "9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b" (UID: "9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b"). InnerVolumeSpecName "kube-api-access-99nxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:54:42.974975 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:42.974940    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99nxn\" (UniqueName: \"kubernetes.io/projected/9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b-kube-api-access-99nxn\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:43.123259 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:43.123172    4864 generic.go:356] "Generic (PLEG): container finished" podID="9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b" containerID="8916621cb9bdd2444ab199e5f6ae7931aa8422cd534e82929bc6bd8a9ba7f794" exitCode=0
Oct 30 01:54:43.123259 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:43.123215    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69"
Oct 30 01:54:43.123453 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:43.123256    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69" event={"ID":"9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b","Type":"ContainerDied","Data":"8916621cb9bdd2444ab199e5f6ae7931aa8422cd534e82929bc6bd8a9ba7f794"}
Oct 30 01:54:43.123453 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:43.123309    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69" event={"ID":"9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b","Type":"ContainerDied","Data":"dc4cce0f24ce705df339a207c2d386821931a795bca74527d7b083b54f89ab6d"}
Oct 30 01:54:43.123453 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:43.123334    4864 scope.go:122] "RemoveContainer" containerID="8916621cb9bdd2444ab199e5f6ae7931aa8422cd534e82929bc6bd8a9ba7f794"
Oct 30 01:54:43.132807 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:43.132792    4864 scope.go:122] "RemoveContainer" containerID="8916621cb9bdd2444ab199e5f6ae7931aa8422cd534e82929bc6bd8a9ba7f794"
Oct 30 01:54:43.133053 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:54:43.133036    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8916621cb9bdd2444ab199e5f6ae7931aa8422cd534e82929bc6bd8a9ba7f794\": container with ID starting with 8916621cb9bdd2444ab199e5f6ae7931aa8422cd534e82929bc6bd8a9ba7f794 not found: ID does not exist" containerID="8916621cb9bdd2444ab199e5f6ae7931aa8422cd534e82929bc6bd8a9ba7f794"
Oct 30 01:54:43.133149 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:43.133061    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8916621cb9bdd2444ab199e5f6ae7931aa8422cd534e82929bc6bd8a9ba7f794"} err="failed to get container status \"8916621cb9bdd2444ab199e5f6ae7931aa8422cd534e82929bc6bd8a9ba7f794\": rpc error: code = NotFound desc = could not find container \"8916621cb9bdd2444ab199e5f6ae7931aa8422cd534e82929bc6bd8a9ba7f794\": container with ID starting with 8916621cb9bdd2444ab199e5f6ae7931aa8422cd534e82929bc6bd8a9ba7f794 not found: ID does not exist"
Oct 30 01:54:43.169704 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:43.169676    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69"]
Oct 30 01:54:43.173229 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:43.173208    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-39b241-69988489d7-cnl69"]
Oct 30 01:54:43.793872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:43.793832    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b" path="/var/lib/kubelet/pods/9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b/volumes"
Oct 30 01:54:49.040144 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:54:49.040104    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:54:51.553055 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:51.553017    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp"]
Oct 30 01:54:51.553586 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:51.553241    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp" podUID="5bf2af8d-0d32-4191-ac7a-69f8aef47e1a" containerName="maas-api" containerID="cri-o://83cd20bdbfa59679eb170fdb679c0e888316266bb8210835fcf01012916ce5c1" gracePeriod=30
Oct 30 01:54:51.836584 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:51.836551    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp"
Oct 30 01:54:51.972860 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:51.972827    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-maas-api-tls\") pod \"5bf2af8d-0d32-4191-ac7a-69f8aef47e1a\" (UID: \"5bf2af8d-0d32-4191-ac7a-69f8aef47e1a\") "
Oct 30 01:54:51.973040 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:51.972877    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-openshift-service-ca\") pod \"5bf2af8d-0d32-4191-ac7a-69f8aef47e1a\" (UID: \"5bf2af8d-0d32-4191-ac7a-69f8aef47e1a\") "
Oct 30 01:54:51.973040 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:51.973000    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-metrics-certs\") pod \"5bf2af8d-0d32-4191-ac7a-69f8aef47e1a\" (UID: \"5bf2af8d-0d32-4191-ac7a-69f8aef47e1a\") "
Oct 30 01:54:51.973165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:51.973148    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-kube-api-access-xl7n6\" (UniqueName: \"kubernetes.io/projected/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-kube-api-access-xl7n6\") pod \"5bf2af8d-0d32-4191-ac7a-69f8aef47e1a\" (UID: \"5bf2af8d-0d32-4191-ac7a-69f8aef47e1a\") "
Oct 30 01:54:51.973294 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:51.973274    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-openshift-service-ca" pod "5bf2af8d-0d32-4191-ac7a-69f8aef47e1a" (UID: "5bf2af8d-0d32-4191-ac7a-69f8aef47e1a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:54:51.973559 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:51.973528    4864 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-openshift-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:51.974963 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:51.974931    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-maas-api-tls" pod "5bf2af8d-0d32-4191-ac7a-69f8aef47e1a" (UID: "5bf2af8d-0d32-4191-ac7a-69f8aef47e1a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:54:51.975322 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:51.975301    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-metrics-certs" pod "5bf2af8d-0d32-4191-ac7a-69f8aef47e1a" (UID: "5bf2af8d-0d32-4191-ac7a-69f8aef47e1a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:54:51.975322 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:51.975306    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-kube-api-access-xl7n6" pod "5bf2af8d-0d32-4191-ac7a-69f8aef47e1a" (UID: "5bf2af8d-0d32-4191-ac7a-69f8aef47e1a"). InnerVolumeSpecName "kube-api-access-xl7n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:54:52.074848 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:52.074769    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-maas-api-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:52.074848 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:52.074798    4864 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-metrics-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:52.074848 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:52.074808    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl7n6\" (UniqueName: \"kubernetes.io/projected/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a-kube-api-access-xl7n6\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:52.175376 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:52.175337    4864 generic.go:356] "Generic (PLEG): container finished" podID="5bf2af8d-0d32-4191-ac7a-69f8aef47e1a" containerID="83cd20bdbfa59679eb170fdb679c0e888316266bb8210835fcf01012916ce5c1" exitCode=0
Oct 30 01:54:52.175541 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:52.175389    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp"
Oct 30 01:54:52.175541 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:52.175432    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp" event={"ID":"5bf2af8d-0d32-4191-ac7a-69f8aef47e1a","Type":"ContainerDied","Data":"83cd20bdbfa59679eb170fdb679c0e888316266bb8210835fcf01012916ce5c1"}
Oct 30 01:54:52.175541 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:52.175500    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp" event={"ID":"5bf2af8d-0d32-4191-ac7a-69f8aef47e1a","Type":"ContainerDied","Data":"1c23455cd339eff0c2b023a2965e7b4479423e70007983aa3904d62ccdb37f02"}
Oct 30 01:54:52.175541 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:52.175530    4864 scope.go:122] "RemoveContainer" containerID="83cd20bdbfa59679eb170fdb679c0e888316266bb8210835fcf01012916ce5c1"
Oct 30 01:54:52.185693 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:52.185673    4864 scope.go:122] "RemoveContainer" containerID="83cd20bdbfa59679eb170fdb679c0e888316266bb8210835fcf01012916ce5c1"
Oct 30 01:54:52.185973 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:54:52.185934    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83cd20bdbfa59679eb170fdb679c0e888316266bb8210835fcf01012916ce5c1\": container with ID starting with 83cd20bdbfa59679eb170fdb679c0e888316266bb8210835fcf01012916ce5c1 not found: ID does not exist" containerID="83cd20bdbfa59679eb170fdb679c0e888316266bb8210835fcf01012916ce5c1"
Oct 30 01:54:52.185973 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:52.185962    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83cd20bdbfa59679eb170fdb679c0e888316266bb8210835fcf01012916ce5c1"} err="failed to get container status \"83cd20bdbfa59679eb170fdb679c0e888316266bb8210835fcf01012916ce5c1\": rpc error: code = NotFound desc = could not find container \"83cd20bdbfa59679eb170fdb679c0e888316266bb8210835fcf01012916ce5c1\": container with ID starting with 83cd20bdbfa59679eb170fdb679c0e888316266bb8210835fcf01012916ce5c1 not found: ID does not exist"
Oct 30 01:54:52.222227 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:52.222197    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp"]
Oct 30 01:54:52.231859 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:52.231826    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-39b241-86f7cdb769-hccfp"]
Oct 30 01:54:53.793592 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:53.793549    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bf2af8d-0d32-4191-ac7a-69f8aef47e1a" path="/var/lib/kubelet/pods/5bf2af8d-0d32-4191-ac7a-69f8aef47e1a/volumes"
Oct 30 01:54:58.017322 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:58.017279    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl"]
Oct 30 01:54:58.017820 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:58.017530    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl" podUID="79ebbfb0-bf32-49c1-8feb-60b55eadbf55" containerName="maas-api" containerID="cri-o://58fc5a522e2fc68e2a088ceaacfcf6ed8d5ad4ab72d28aa0cfd9937338cdb14e" gracePeriod=30
Oct 30 01:54:58.217785 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:58.217745    4864 generic.go:356] "Generic (PLEG): container finished" podID="79ebbfb0-bf32-49c1-8feb-60b55eadbf55" containerID="58fc5a522e2fc68e2a088ceaacfcf6ed8d5ad4ab72d28aa0cfd9937338cdb14e" exitCode=0
Oct 30 01:54:58.218017 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:58.217794    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl" event={"ID":"79ebbfb0-bf32-49c1-8feb-60b55eadbf55","Type":"ContainerDied","Data":"58fc5a522e2fc68e2a088ceaacfcf6ed8d5ad4ab72d28aa0cfd9937338cdb14e"}
Oct 30 01:54:58.303491 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:58.303460    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl"
Oct 30 01:54:58.346369 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:58.346332    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-maas-api-tls\") pod \"79ebbfb0-bf32-49c1-8feb-60b55eadbf55\" (UID: \"79ebbfb0-bf32-49c1-8feb-60b55eadbf55\") "
Oct 30 01:54:58.346545 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:58.346402    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-metrics-certs\") pod \"79ebbfb0-bf32-49c1-8feb-60b55eadbf55\" (UID: \"79ebbfb0-bf32-49c1-8feb-60b55eadbf55\") "
Oct 30 01:54:58.346545 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:58.346508    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-kube-api-access-kxcxj\" (UniqueName: \"kubernetes.io/projected/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-kube-api-access-kxcxj\") pod \"79ebbfb0-bf32-49c1-8feb-60b55eadbf55\" (UID: \"79ebbfb0-bf32-49c1-8feb-60b55eadbf55\") "
Oct 30 01:54:58.346652 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:58.346588    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-openshift-service-ca\") pod \"79ebbfb0-bf32-49c1-8feb-60b55eadbf55\" (UID: \"79ebbfb0-bf32-49c1-8feb-60b55eadbf55\") "
Oct 30 01:54:58.347036 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:58.347005    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-openshift-service-ca" pod "79ebbfb0-bf32-49c1-8feb-60b55eadbf55" (UID: "79ebbfb0-bf32-49c1-8feb-60b55eadbf55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:54:58.349056 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:58.349033    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-kube-api-access-kxcxj" pod "79ebbfb0-bf32-49c1-8feb-60b55eadbf55" (UID: "79ebbfb0-bf32-49c1-8feb-60b55eadbf55"). InnerVolumeSpecName "kube-api-access-kxcxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:54:58.349179 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:58.349107    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-metrics-certs" pod "79ebbfb0-bf32-49c1-8feb-60b55eadbf55" (UID: "79ebbfb0-bf32-49c1-8feb-60b55eadbf55"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:54:58.349179 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:58.349118    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-maas-api-tls" pod "79ebbfb0-bf32-49c1-8feb-60b55eadbf55" (UID: "79ebbfb0-bf32-49c1-8feb-60b55eadbf55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:54:58.451161 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:58.451124    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxcxj\" (UniqueName: \"kubernetes.io/projected/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-kube-api-access-kxcxj\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:58.451161 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:58.451163    4864 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-openshift-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:58.451378 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:58.451178    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-maas-api-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:58.451378 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:58.451190    4864 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ebbfb0-bf32-49c1-8feb-60b55eadbf55-metrics-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:54:59.225518 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:59.225490    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl"
Oct 30 01:54:59.225948 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:59.225513    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl" event={"ID":"79ebbfb0-bf32-49c1-8feb-60b55eadbf55","Type":"ContainerDied","Data":"e20aa467724e3956dfb085989ccc540ddf0e9786625a3cacd32518bc027a355d"}
Oct 30 01:54:59.225948 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:59.225578    4864 scope.go:122] "RemoveContainer" containerID="58fc5a522e2fc68e2a088ceaacfcf6ed8d5ad4ab72d28aa0cfd9937338cdb14e"
Oct 30 01:54:59.276579 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:59.276539    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl"]
Oct 30 01:54:59.287806 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:59.287769    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9da49f-664ddb98b4-8fqdl"]
Oct 30 01:54:59.794399 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:54:59.794364    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79ebbfb0-bf32-49c1-8feb-60b55eadbf55" path="/var/lib/kubelet/pods/79ebbfb0-bf32-49c1-8feb-60b55eadbf55/volumes"
Oct 30 01:55:05.882992 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:05.882958    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"]
Oct 30 01:55:05.883409 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:05.883194    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk" podUID="3a94e905-9533-4db9-b8d2-5c10538e0d0b" containerName="istio-proxy" containerID="cri-o://ed07846e19d4ade2537dd402e5b7fe5932e72263c716f8f2036101b74d8854f0" gracePeriod=30
Oct 30 01:55:08.661324 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:08.661284    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"]
Oct 30 01:55:08.661752 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:08.661538    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz" podUID="ddfb64f0-d446-48ec-90e7-6f2c1e36920f" containerName="istio-proxy" containerID="cri-o://0654d8e852974a70df6babbea8b16909d1bbd281129e449eb6819a656356c7d4" gracePeriod=30
Oct 30 01:55:09.126642 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:09.126597    4864 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:55:09.126793 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:09.126655    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz" podUID="ddfb64f0-d446-48ec-90e7-6f2c1e36920f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:55:11.159714 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.159645    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:55:11.287581 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.287542    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-workload-socket\") pod \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") "
Oct 30 01:55:11.287781 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.287588    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-podinfo\") pod \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") "
Oct 30 01:55:11.287781 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.287612    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a94e905-9533-4db9-b8d2-5c10538e0d0b-kube-api-access-z7bwr\" (UniqueName: \"kubernetes.io/projected/3a94e905-9533-4db9-b8d2-5c10538e0d0b-kube-api-access-z7bwr\") pod \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") "
Oct 30 01:55:11.287781 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.287674    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-data\") pod \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") "
Oct 30 01:55:11.287781 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.287710    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-credential-socket\") pod \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") "
Oct 30 01:55:11.287781 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.287775    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-envoy\") pod \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") "
Oct 30 01:55:11.288106 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.287862    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istiod-ca-cert\") pod \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") "
Oct 30 01:55:11.288106 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.287870    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-workload-socket" pod "3a94e905-9533-4db9-b8d2-5c10538e0d0b" (UID: "3a94e905-9533-4db9-b8d2-5c10538e0d0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:11.288106 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.287907    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-workload-certs\") pod \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") "
Oct 30 01:55:11.288106 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.287967    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-token\" (UniqueName: \"kubernetes.io/projected/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-token\") pod \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\" (UID: \"3a94e905-9533-4db9-b8d2-5c10538e0d0b\") "
Oct 30 01:55:11.288351 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.288110    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-data" pod "3a94e905-9533-4db9-b8d2-5c10538e0d0b" (UID: "3a94e905-9533-4db9-b8d2-5c10538e0d0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:11.288412 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.288142    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-credential-socket" pod "3a94e905-9533-4db9-b8d2-5c10538e0d0b" (UID: "3a94e905-9533-4db9-b8d2-5c10538e0d0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:11.288412 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.288295    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istiod-ca-cert" pod "3a94e905-9533-4db9-b8d2-5c10538e0d0b" (UID: "3a94e905-9533-4db9-b8d2-5c10538e0d0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:11.288529 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.288377    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-workload-certs" pod "3a94e905-9533-4db9-b8d2-5c10538e0d0b" (UID: "3a94e905-9533-4db9-b8d2-5c10538e0d0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:11.288529 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.288519    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:11.288638 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.288541    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:11.288638 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.288554    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:11.288638 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.288567    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:11.288638 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.288580    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:11.290155 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.290126    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-podinfo" pod "3a94e905-9533-4db9-b8d2-5c10538e0d0b" (UID: "3a94e905-9533-4db9-b8d2-5c10538e0d0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:55:11.290298 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.290149    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a94e905-9533-4db9-b8d2-5c10538e0d0b-kube-api-access-z7bwr" pod "3a94e905-9533-4db9-b8d2-5c10538e0d0b" (UID: "3a94e905-9533-4db9-b8d2-5c10538e0d0b"). InnerVolumeSpecName "kube-api-access-z7bwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:11.290298 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.290281    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-envoy" pod "3a94e905-9533-4db9-b8d2-5c10538e0d0b" (UID: "3a94e905-9533-4db9-b8d2-5c10538e0d0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:11.290395 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.290294    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-token" pod "3a94e905-9533-4db9-b8d2-5c10538e0d0b" (UID: "3a94e905-9533-4db9-b8d2-5c10538e0d0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:11.297467 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.297438    4864 generic.go:356] "Generic (PLEG): container finished" podID="3a94e905-9533-4db9-b8d2-5c10538e0d0b" containerID="ed07846e19d4ade2537dd402e5b7fe5932e72263c716f8f2036101b74d8854f0" exitCode=0
Oct 30 01:55:11.297572 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.297488    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"
Oct 30 01:55:11.297572 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.297498    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk" event={"ID":"3a94e905-9533-4db9-b8d2-5c10538e0d0b","Type":"ContainerDied","Data":"ed07846e19d4ade2537dd402e5b7fe5932e72263c716f8f2036101b74d8854f0"}
Oct 30 01:55:11.297650 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.297587    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk" event={"ID":"3a94e905-9533-4db9-b8d2-5c10538e0d0b","Type":"ContainerDied","Data":"5d70542fe175ea94442c43fa30454e69dc5bd33705fc2c1800f0ba8426115792"}
Oct 30 01:55:11.297650 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.297617    4864 scope.go:122] "RemoveContainer" containerID="ed07846e19d4ade2537dd402e5b7fe5932e72263c716f8f2036101b74d8854f0"
Oct 30 01:55:11.315742 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.315722    4864 scope.go:122] "RemoveContainer" containerID="e9406d9dc7f85a9a274b160d43844a46acf193bd2ed252ccb1428d25316824e2"
Oct 30 01:55:11.324496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.324476    4864 scope.go:122] "RemoveContainer" containerID="ed07846e19d4ade2537dd402e5b7fe5932e72263c716f8f2036101b74d8854f0"
Oct 30 01:55:11.324766 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:55:11.324745    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed07846e19d4ade2537dd402e5b7fe5932e72263c716f8f2036101b74d8854f0\": container with ID starting with ed07846e19d4ade2537dd402e5b7fe5932e72263c716f8f2036101b74d8854f0 not found: ID does not exist" containerID="ed07846e19d4ade2537dd402e5b7fe5932e72263c716f8f2036101b74d8854f0"
Oct 30 01:55:11.324844 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.324780    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed07846e19d4ade2537dd402e5b7fe5932e72263c716f8f2036101b74d8854f0"} err="failed to get container status \"ed07846e19d4ade2537dd402e5b7fe5932e72263c716f8f2036101b74d8854f0\": rpc error: code = NotFound desc = could not find container \"ed07846e19d4ade2537dd402e5b7fe5932e72263c716f8f2036101b74d8854f0\": container with ID starting with ed07846e19d4ade2537dd402e5b7fe5932e72263c716f8f2036101b74d8854f0 not found: ID does not exist"
Oct 30 01:55:11.324844 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.324805    4864 scope.go:122] "RemoveContainer" containerID="e9406d9dc7f85a9a274b160d43844a46acf193bd2ed252ccb1428d25316824e2"
Oct 30 01:55:11.325065 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:55:11.325040    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9406d9dc7f85a9a274b160d43844a46acf193bd2ed252ccb1428d25316824e2\": container with ID starting with e9406d9dc7f85a9a274b160d43844a46acf193bd2ed252ccb1428d25316824e2 not found: ID does not exist" containerID="e9406d9dc7f85a9a274b160d43844a46acf193bd2ed252ccb1428d25316824e2"
Oct 30 01:55:11.325131 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.325092    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9406d9dc7f85a9a274b160d43844a46acf193bd2ed252ccb1428d25316824e2"} err="failed to get container status \"e9406d9dc7f85a9a274b160d43844a46acf193bd2ed252ccb1428d25316824e2\": rpc error: code = NotFound desc = could not find container \"e9406d9dc7f85a9a274b160d43844a46acf193bd2ed252ccb1428d25316824e2\": container with ID starting with e9406d9dc7f85a9a274b160d43844a46acf193bd2ed252ccb1428d25316824e2 not found: ID does not exist"
Oct 30 01:55:11.352251 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.352220    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"]
Oct 30 01:55:11.359987 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.359960    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1854c4-openshift-default-d45469cc-ltftk"]
Oct 30 01:55:11.389597 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.389565    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:11.389597 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.389592    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:11.389597 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.389602    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7bwr\" (UniqueName: \"kubernetes.io/projected/3a94e905-9533-4db9-b8d2-5c10538e0d0b-kube-api-access-z7bwr\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:11.389801 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.389612    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a94e905-9533-4db9-b8d2-5c10538e0d0b-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:11.793345 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:11.793305    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a94e905-9533-4db9-b8d2-5c10538e0d0b" path="/var/lib/kubelet/pods/3a94e905-9533-4db9-b8d2-5c10538e0d0b/volumes"
Oct 30 01:55:13.015840 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:13.015805    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"]
Oct 30 01:55:13.026446 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:13.026410    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"]
Oct 30 01:55:13.919535 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:13.919508    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:55:14.015773 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.015742    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-workload-socket\") pod \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") "
Oct 30 01:55:14.015773 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.015777    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-credential-socket\") pod \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") "
Oct 30 01:55:14.016390 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.015837    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istiod-ca-cert\") pod \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") "
Oct 30 01:55:14.016390 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.015874    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-token\" (UniqueName: \"kubernetes.io/projected/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-token\") pod \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") "
Oct 30 01:55:14.016390 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.015895    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-workload-certs\") pod \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") "
Oct 30 01:55:14.016390 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.015946    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-data\") pod \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") "
Oct 30 01:55:14.016390 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.015990    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-envoy\") pod \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") "
Oct 30 01:55:14.016390 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.016081    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-workload-socket" pod "ddfb64f0-d446-48ec-90e7-6f2c1e36920f" (UID: "ddfb64f0-d446-48ec-90e7-6f2c1e36920f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:14.016390 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.016105    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-kube-api-access-77fvv\" (UniqueName: \"kubernetes.io/projected/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-kube-api-access-77fvv\") pod \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") "
Oct 30 01:55:14.016390 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.016171    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-podinfo\") pod \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\" (UID: \"ddfb64f0-d446-48ec-90e7-6f2c1e36920f\") "
Oct 30 01:55:14.016390 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.016200    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-credential-socket" pod "ddfb64f0-d446-48ec-90e7-6f2c1e36920f" (UID: "ddfb64f0-d446-48ec-90e7-6f2c1e36920f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:14.016390 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.016306    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-data" pod "ddfb64f0-d446-48ec-90e7-6f2c1e36920f" (UID: "ddfb64f0-d446-48ec-90e7-6f2c1e36920f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:14.016390 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.016341    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istiod-ca-cert" pod "ddfb64f0-d446-48ec-90e7-6f2c1e36920f" (UID: "ddfb64f0-d446-48ec-90e7-6f2c1e36920f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:14.016857 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.016594    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-workload-certs" pod "ddfb64f0-d446-48ec-90e7-6f2c1e36920f" (UID: "ddfb64f0-d446-48ec-90e7-6f2c1e36920f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:14.016857 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.016600    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:14.016857 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.016634    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:14.016857 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.016650    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:14.016857 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.016665    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:14.018680 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.018656    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-podinfo" pod "ddfb64f0-d446-48ec-90e7-6f2c1e36920f" (UID: "ddfb64f0-d446-48ec-90e7-6f2c1e36920f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:55:14.018795 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.018703    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-token" pod "ddfb64f0-d446-48ec-90e7-6f2c1e36920f" (UID: "ddfb64f0-d446-48ec-90e7-6f2c1e36920f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:14.018795 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.018786    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-kube-api-access-77fvv" pod "ddfb64f0-d446-48ec-90e7-6f2c1e36920f" (UID: "ddfb64f0-d446-48ec-90e7-6f2c1e36920f"). InnerVolumeSpecName "kube-api-access-77fvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:14.019025 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.018998    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-envoy" pod "ddfb64f0-d446-48ec-90e7-6f2c1e36920f" (UID: "ddfb64f0-d446-48ec-90e7-6f2c1e36920f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:14.117652 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.117621    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:14.117652 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.117647    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:14.117652 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.117657    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:14.117887 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.117666    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77fvv\" (UniqueName: \"kubernetes.io/projected/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-kube-api-access-77fvv\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:14.117887 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.117675    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddfb64f0-d446-48ec-90e7-6f2c1e36920f-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:14.316990 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.316890    4864 generic.go:356] "Generic (PLEG): container finished" podID="ddfb64f0-d446-48ec-90e7-6f2c1e36920f" containerID="0654d8e852974a70df6babbea8b16909d1bbd281129e449eb6819a656356c7d4" exitCode=0
Oct 30 01:55:14.316990 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.316946    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"
Oct 30 01:55:14.317277 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.316979    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz" event={"ID":"ddfb64f0-d446-48ec-90e7-6f2c1e36920f","Type":"ContainerDied","Data":"0654d8e852974a70df6babbea8b16909d1bbd281129e449eb6819a656356c7d4"}
Oct 30 01:55:14.317277 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.317043    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz" event={"ID":"ddfb64f0-d446-48ec-90e7-6f2c1e36920f","Type":"ContainerDied","Data":"91d952f086a3d4a864708c81cb0338388e5a83a8fddfd10f9f426bcde669e314"}
Oct 30 01:55:14.317277 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.317098    4864 scope.go:122] "RemoveContainer" containerID="0654d8e852974a70df6babbea8b16909d1bbd281129e449eb6819a656356c7d4"
Oct 30 01:55:14.327594 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.327577    4864 scope.go:122] "RemoveContainer" containerID="0654d8e852974a70df6babbea8b16909d1bbd281129e449eb6819a656356c7d4"
Oct 30 01:55:14.327906 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:55:14.327885    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0654d8e852974a70df6babbea8b16909d1bbd281129e449eb6819a656356c7d4\": container with ID starting with 0654d8e852974a70df6babbea8b16909d1bbd281129e449eb6819a656356c7d4 not found: ID does not exist" containerID="0654d8e852974a70df6babbea8b16909d1bbd281129e449eb6819a656356c7d4"
Oct 30 01:55:14.327963 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.327915    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0654d8e852974a70df6babbea8b16909d1bbd281129e449eb6819a656356c7d4"} err="failed to get container status \"0654d8e852974a70df6babbea8b16909d1bbd281129e449eb6819a656356c7d4\": rpc error: code = NotFound desc = could not find container \"0654d8e852974a70df6babbea8b16909d1bbd281129e449eb6819a656356c7d4\": container with ID starting with 0654d8e852974a70df6babbea8b16909d1bbd281129e449eb6819a656356c7d4 not found: ID does not exist"
Oct 30 01:55:14.365257 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.365227    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"]
Oct 30 01:55:14.376010 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:14.375984    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8ae41e-openshift-default-6775d5944c-hgzkz"]
Oct 30 01:55:15.793248 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:15.793213    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddfb64f0-d446-48ec-90e7-6f2c1e36920f" path="/var/lib/kubelet/pods/ddfb64f0-d446-48ec-90e7-6f2c1e36920f/volumes"
Oct 30 01:55:19.038949 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:55:19.038911    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:55:28.037656 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.037599    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz" podUID="86e98e6a-b1cf-4d08-b807-26397cd1b4a4" containerName="main" containerID="cri-o://de65ecc9869bae257c2ab7b41545bd61b89f9e541d446d1e8fec352138b61dc5" gracePeriod=45
Oct 30 01:55:28.046104 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.046057    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29" podUID="e0ab9bcb-f48a-404b-b787-9f8c43b58722" containerName="main" containerID="cri-o://25fb162a342fdc5a92afe2f784dfc6265c1ee913d8bc226a9b302e4ff3c1c7e0" gracePeriod=45
Oct 30 01:55:28.323827 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.323801    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 01:55:28.326928 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.326908    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 01:55:28.402772 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.402735    4864 generic.go:356] "Generic (PLEG): container finished" podID="e0ab9bcb-f48a-404b-b787-9f8c43b58722" containerID="25fb162a342fdc5a92afe2f784dfc6265c1ee913d8bc226a9b302e4ff3c1c7e0" exitCode=0
Oct 30 01:55:28.402971 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.402785    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"
Oct 30 01:55:28.402971 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.402887    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29" event={"ID":"e0ab9bcb-f48a-404b-b787-9f8c43b58722","Type":"ContainerDied","Data":"25fb162a342fdc5a92afe2f784dfc6265c1ee913d8bc226a9b302e4ff3c1c7e0"}
Oct 30 01:55:28.402971 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.402950    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29" event={"ID":"e0ab9bcb-f48a-404b-b787-9f8c43b58722","Type":"ContainerDied","Data":"5131dee2fd77ada61c6f20db470ba46b7c6ccef1be2d5770c08348eff14ee79e"}
Oct 30 01:55:28.403205 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.402978    4864 scope.go:122] "RemoveContainer" containerID="25fb162a342fdc5a92afe2f784dfc6265c1ee913d8bc226a9b302e4ff3c1c7e0"
Oct 30 01:55:28.405551 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.405387    4864 generic.go:356] "Generic (PLEG): container finished" podID="86e98e6a-b1cf-4d08-b807-26397cd1b4a4" containerID="de65ecc9869bae257c2ab7b41545bd61b89f9e541d446d1e8fec352138b61dc5" exitCode=0
Oct 30 01:55:28.405551 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.405428    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"
Oct 30 01:55:28.405551 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.405477    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz" event={"ID":"86e98e6a-b1cf-4d08-b807-26397cd1b4a4","Type":"ContainerDied","Data":"de65ecc9869bae257c2ab7b41545bd61b89f9e541d446d1e8fec352138b61dc5"}
Oct 30 01:55:28.405551 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.405507    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz" event={"ID":"86e98e6a-b1cf-4d08-b807-26397cd1b4a4","Type":"ContainerDied","Data":"a9d504d64e9c28ec0e3fade134f084be2bcb5591fdb1e6cf43cd00b18028db6c"}
Oct 30 01:55:28.417756 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.417729    4864 scope.go:122] "RemoveContainer" containerID="25fb162a342fdc5a92afe2f784dfc6265c1ee913d8bc226a9b302e4ff3c1c7e0"
Oct 30 01:55:28.418114 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:55:28.418067    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25fb162a342fdc5a92afe2f784dfc6265c1ee913d8bc226a9b302e4ff3c1c7e0\": container with ID starting with 25fb162a342fdc5a92afe2f784dfc6265c1ee913d8bc226a9b302e4ff3c1c7e0 not found: ID does not exist" containerID="25fb162a342fdc5a92afe2f784dfc6265c1ee913d8bc226a9b302e4ff3c1c7e0"
Oct 30 01:55:28.418231 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.418125    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25fb162a342fdc5a92afe2f784dfc6265c1ee913d8bc226a9b302e4ff3c1c7e0"} err="failed to get container status \"25fb162a342fdc5a92afe2f784dfc6265c1ee913d8bc226a9b302e4ff3c1c7e0\": rpc error: code = NotFound desc = could not find container \"25fb162a342fdc5a92afe2f784dfc6265c1ee913d8bc226a9b302e4ff3c1c7e0\": container with ID starting with 25fb162a342fdc5a92afe2f784dfc6265c1ee913d8bc226a9b302e4ff3c1c7e0 not found: ID does not exist"
Oct 30 01:55:28.418231 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.418148    4864 scope.go:122] "RemoveContainer" containerID="de65ecc9869bae257c2ab7b41545bd61b89f9e541d446d1e8fec352138b61dc5"
Oct 30 01:55:28.428555 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.428532    4864 scope.go:122] "RemoveContainer" containerID="de65ecc9869bae257c2ab7b41545bd61b89f9e541d446d1e8fec352138b61dc5"
Oct 30 01:55:28.428853 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:55:28.428831    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de65ecc9869bae257c2ab7b41545bd61b89f9e541d446d1e8fec352138b61dc5\": container with ID starting with de65ecc9869bae257c2ab7b41545bd61b89f9e541d446d1e8fec352138b61dc5 not found: ID does not exist" containerID="de65ecc9869bae257c2ab7b41545bd61b89f9e541d446d1e8fec352138b61dc5"
Oct 30 01:55:28.428941 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.428867    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de65ecc9869bae257c2ab7b41545bd61b89f9e541d446d1e8fec352138b61dc5"} err="failed to get container status \"de65ecc9869bae257c2ab7b41545bd61b89f9e541d446d1e8fec352138b61dc5\": rpc error: code = NotFound desc = could not find container \"de65ecc9869bae257c2ab7b41545bd61b89f9e541d446d1e8fec352138b61dc5\": container with ID starting with de65ecc9869bae257c2ab7b41545bd61b89f9e541d446d1e8fec352138b61dc5 not found: ID does not exist"
Oct 30 01:55:28.474494 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.474454    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0ab9bcb-f48a-404b-b787-9f8c43b58722-tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ab9bcb-f48a-404b-b787-9f8c43b58722-tls-certs\") pod \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\" (UID: \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\") "
Oct 30 01:55:28.474665 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.474538    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-tls-certs\" (UniqueName: \"kubernetes.io/secret/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-tls-certs\") pod \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\" (UID: \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\") "
Oct 30 01:55:28.474665 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.474574    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-dshm\") pod \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\" (UID: \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\") "
Oct 30 01:55:28.474665 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.474616    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-dshm\") pod \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\" (UID: \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\") "
Oct 30 01:55:28.474665 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.474645    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-home\" (UniqueName: \"kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-home\") pod \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\" (UID: \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\") "
Oct 30 01:55:28.474901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.474681    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-kube-api-access-x8dfm\" (UniqueName: \"kubernetes.io/projected/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-kube-api-access-x8dfm\") pod \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\" (UID: \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\") "
Oct 30 01:55:28.474901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.474714    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-tmp-dir\") pod \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\" (UID: \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\") "
Oct 30 01:55:28.474901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.474770    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-model-cache\") pod \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\" (UID: \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\") "
Oct 30 01:55:28.474901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.474813    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-model-cache\") pod \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\" (UID: \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\") "
Oct 30 01:55:28.474901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.474861    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-tmp-dir\") pod \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\" (UID: \"86e98e6a-b1cf-4d08-b807-26397cd1b4a4\") "
Oct 30 01:55:28.475206 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.474931    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-home\" (UniqueName: \"kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-home\") pod \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\" (UID: \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\") "
Oct 30 01:55:28.475206 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.474963    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ab9bcb-f48a-404b-b787-9f8c43b58722-kube-api-access-6jvgd\" (UniqueName: \"kubernetes.io/projected/e0ab9bcb-f48a-404b-b787-9f8c43b58722-kube-api-access-6jvgd\") pod \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\" (UID: \"e0ab9bcb-f48a-404b-b787-9f8c43b58722\") "
Oct 30 01:55:28.475303 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.475274    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-model-cache" pod "86e98e6a-b1cf-4d08-b807-26397cd1b4a4" (UID: "86e98e6a-b1cf-4d08-b807-26397cd1b4a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:28.475427 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.475405    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-home" pod "86e98e6a-b1cf-4d08-b807-26397cd1b4a4" (UID: "86e98e6a-b1cf-4d08-b807-26397cd1b4a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:28.475500 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.475420    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-model-cache" pod "e0ab9bcb-f48a-404b-b787-9f8c43b58722" (UID: "e0ab9bcb-f48a-404b-b787-9f8c43b58722"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:28.475500 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.475459    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-tmp-dir" pod "e0ab9bcb-f48a-404b-b787-9f8c43b58722" (UID: "e0ab9bcb-f48a-404b-b787-9f8c43b58722"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:28.475500 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.475472    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-tmp-dir" pod "86e98e6a-b1cf-4d08-b807-26397cd1b4a4" (UID: "86e98e6a-b1cf-4d08-b807-26397cd1b4a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:28.475778 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.475758    4864 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-home\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:28.475866 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.475782    4864 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-tmp-dir\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:28.475866 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.475798    4864 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-model-cache\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:28.475866 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.475810    4864 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-model-cache\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:28.475866 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.475822    4864 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-tmp-dir\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:28.476200 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.475877    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-home" pod "e0ab9bcb-f48a-404b-b787-9f8c43b58722" (UID: "e0ab9bcb-f48a-404b-b787-9f8c43b58722"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:28.477429 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.477403    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-dshm" pod "86e98e6a-b1cf-4d08-b807-26397cd1b4a4" (UID: "86e98e6a-b1cf-4d08-b807-26397cd1b4a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:28.477530 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.477456    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0ab9bcb-f48a-404b-b787-9f8c43b58722-tls-certs" pod "e0ab9bcb-f48a-404b-b787-9f8c43b58722" (UID: "e0ab9bcb-f48a-404b-b787-9f8c43b58722"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:28.477530 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.477486    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-tls-certs" pod "86e98e6a-b1cf-4d08-b807-26397cd1b4a4" (UID: "86e98e6a-b1cf-4d08-b807-26397cd1b4a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:28.477649 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.477617    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-dshm" pod "e0ab9bcb-f48a-404b-b787-9f8c43b58722" (UID: "e0ab9bcb-f48a-404b-b787-9f8c43b58722"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:28.477810 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.477790    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ab9bcb-f48a-404b-b787-9f8c43b58722-kube-api-access-6jvgd" pod "e0ab9bcb-f48a-404b-b787-9f8c43b58722" (UID: "e0ab9bcb-f48a-404b-b787-9f8c43b58722"). InnerVolumeSpecName "kube-api-access-6jvgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:28.478050 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.478031    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-kube-api-access-x8dfm" pod "86e98e6a-b1cf-4d08-b807-26397cd1b4a4" (UID: "86e98e6a-b1cf-4d08-b807-26397cd1b4a4"). InnerVolumeSpecName "kube-api-access-x8dfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:28.577489 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.577406    4864 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-tls-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:28.577489 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.577440    4864 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-dshm\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:28.577489 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.577453    4864 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-dshm\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:28.577489 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.577461    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8dfm\" (UniqueName: \"kubernetes.io/projected/86e98e6a-b1cf-4d08-b807-26397cd1b4a4-kube-api-access-x8dfm\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:28.577489 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.577469    4864 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ab9bcb-f48a-404b-b787-9f8c43b58722-home\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:28.577489 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.577478    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jvgd\" (UniqueName: \"kubernetes.io/projected/e0ab9bcb-f48a-404b-b787-9f8c43b58722-kube-api-access-6jvgd\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:28.577489 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.577487    4864 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ab9bcb-f48a-404b-b787-9f8c43b58722-tls-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:28.744380 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.744350    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"]
Oct 30 01:55:28.754384 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.754356    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-72ed13/distinct-w2-72ed13-kserve-b8d9b5dc5-pgs29"]
Oct 30 01:55:28.778111 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.778083    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"]
Oct 30 01:55:28.783601 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:28.783576    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz"]
Oct 30 01:55:29.221801 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.221749    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/distinct-2-w2-72ed13-kserve-8676f55c7b-vxcrz" podUID="86e98e6a-b1cf-4d08-b807-26397cd1b4a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:55:29.389332 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389251    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"]
Oct 30 01:55:29.389760 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389742    4864 cpu_manager.go:414] "containerMap: removing container" podUID="89b1105d-0395-4e80-a63b-7e8e0cce276d" containerName="istio-proxy"
Oct 30 01:55:29.389760 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389759    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b1105d-0395-4e80-a63b-7e8e0cce276d" containerName="istio-proxy"
Oct 30 01:55:29.389902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389773    4864 cpu_manager.go:414] "containerMap: removing container" podUID="ac76439d-8c30-4810-b242-1a3e1ed2b556" containerName="maas-api"
Oct 30 01:55:29.389902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389779    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac76439d-8c30-4810-b242-1a3e1ed2b556" containerName="maas-api"
Oct 30 01:55:29.389902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389787    4864 cpu_manager.go:414] "containerMap: removing container" podUID="778281fb-fe1b-4f27-bd43-62920826a0f9" containerName="istio-proxy"
Oct 30 01:55:29.389902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389792    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="778281fb-fe1b-4f27-bd43-62920826a0f9" containerName="istio-proxy"
Oct 30 01:55:29.389902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389800    4864 cpu_manager.go:414] "containerMap: removing container" podUID="afd0afda-48df-44ec-bebd-52768b7bdbc0" containerName="istio-proxy"
Oct 30 01:55:29.389902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389805    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd0afda-48df-44ec-bebd-52768b7bdbc0" containerName="istio-proxy"
Oct 30 01:55:29.389902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389811    4864 cpu_manager.go:414] "containerMap: removing container" podUID="afd0afda-48df-44ec-bebd-52768b7bdbc0" containerName="istio-proxy"
Oct 30 01:55:29.389902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389815    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd0afda-48df-44ec-bebd-52768b7bdbc0" containerName="istio-proxy"
Oct 30 01:55:29.389902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389825    4864 cpu_manager.go:414] "containerMap: removing container" podUID="63a8525d-9951-4f19-b124-62e4fa91c6b9" containerName="main"
Oct 30 01:55:29.389902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389830    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a8525d-9951-4f19-b124-62e4fa91c6b9" containerName="main"
Oct 30 01:55:29.389902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389837    4864 cpu_manager.go:414] "containerMap: removing container" podUID="e0ab9bcb-f48a-404b-b787-9f8c43b58722" containerName="main"
Oct 30 01:55:29.389902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389841    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ab9bcb-f48a-404b-b787-9f8c43b58722" containerName="main"
Oct 30 01:55:29.389902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389848    4864 cpu_manager.go:414] "containerMap: removing container" podUID="a87828aa-8038-4643-834a-bbdb78432874" containerName="main"
Oct 30 01:55:29.389902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389852    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a87828aa-8038-4643-834a-bbdb78432874" containerName="main"
Oct 30 01:55:29.389902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389861    4864 cpu_manager.go:414] "containerMap: removing container" podUID="55dbdf6b-c1f2-4e0a-907a-7cd59c30e989" containerName="main"
Oct 30 01:55:29.389902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389865    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55dbdf6b-c1f2-4e0a-907a-7cd59c30e989" containerName="main"
Oct 30 01:55:29.389902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389874    4864 cpu_manager.go:414] "containerMap: removing container" podUID="9de67abe-4f7c-441f-9e3e-cf34c74068d9" containerName="istio-proxy"
Oct 30 01:55:29.389902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389878    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9de67abe-4f7c-441f-9e3e-cf34c74068d9" containerName="istio-proxy"
Oct 30 01:55:29.389902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389891    4864 cpu_manager.go:414] "containerMap: removing container" podUID="8b267884-baae-4ad3-a414-82467f7188b3" containerName="main"
Oct 30 01:55:29.389902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389896    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b267884-baae-4ad3-a414-82467f7188b3" containerName="main"
Oct 30 01:55:29.389902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389902    4864 cpu_manager.go:414] "containerMap: removing container" podUID="86e98e6a-b1cf-4d08-b807-26397cd1b4a4" containerName="main"
Oct 30 01:55:29.389902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389907    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86e98e6a-b1cf-4d08-b807-26397cd1b4a4" containerName="main"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389916    4864 cpu_manager.go:414] "containerMap: removing container" podUID="d0822059-3672-4595-967c-0796222fd966" containerName="payload-pre-processing"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389921    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0822059-3672-4595-967c-0796222fd966" containerName="payload-pre-processing"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389929    4864 cpu_manager.go:414] "containerMap: removing container" podUID="bc75808d-fb11-4820-939a-18037b0c2791" containerName="istio-proxy"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389933    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc75808d-fb11-4820-939a-18037b0c2791" containerName="istio-proxy"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389940    4864 cpu_manager.go:414] "containerMap: removing container" podUID="e92bdf87-b7d6-4060-9615-70ae77214bc0" containerName="istio-proxy"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389944    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e92bdf87-b7d6-4060-9615-70ae77214bc0" containerName="istio-proxy"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389953    4864 cpu_manager.go:414] "containerMap: removing container" podUID="9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b" containerName="payload-pre-processing"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389957    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b" containerName="payload-pre-processing"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389966    4864 cpu_manager.go:414] "containerMap: removing container" podUID="a11b8ed4-c24c-4919-959f-86cf8ecbacb2" containerName="istio-proxy"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389971    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a11b8ed4-c24c-4919-959f-86cf8ecbacb2" containerName="istio-proxy"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389978    4864 cpu_manager.go:414] "containerMap: removing container" podUID="5bf2af8d-0d32-4191-ac7a-69f8aef47e1a" containerName="maas-api"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389982    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bf2af8d-0d32-4191-ac7a-69f8aef47e1a" containerName="maas-api"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389989    4864 cpu_manager.go:414] "containerMap: removing container" podUID="9f728c20-eae2-4fd4-aa40-4094a75f5f16" containerName="maas-api"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389994    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f728c20-eae2-4fd4-aa40-4094a75f5f16" containerName="maas-api"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.389999    4864 cpu_manager.go:414] "containerMap: removing container" podUID="79ebbfb0-bf32-49c1-8feb-60b55eadbf55" containerName="maas-api"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390003    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ebbfb0-bf32-49c1-8feb-60b55eadbf55" containerName="maas-api"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390010    4864 cpu_manager.go:414] "containerMap: removing container" podUID="ddfb64f0-d446-48ec-90e7-6f2c1e36920f" containerName="istio-proxy"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390014    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddfb64f0-d446-48ec-90e7-6f2c1e36920f" containerName="istio-proxy"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390022    4864 cpu_manager.go:414] "containerMap: removing container" podUID="3a94e905-9533-4db9-b8d2-5c10538e0d0b" containerName="istio-proxy"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390027    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a94e905-9533-4db9-b8d2-5c10538e0d0b" containerName="istio-proxy"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390035    4864 cpu_manager.go:414] "containerMap: removing container" podUID="2752235c-68d2-4be4-b1eb-d851d2b5c903" containerName="maas-api"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390039    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2752235c-68d2-4be4-b1eb-d851d2b5c903" containerName="maas-api"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390046    4864 cpu_manager.go:414] "containerMap: removing container" podUID="3a94e905-9533-4db9-b8d2-5c10538e0d0b" containerName="istio-proxy"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390050    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a94e905-9533-4db9-b8d2-5c10538e0d0b" containerName="istio-proxy"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390156    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b267884-baae-4ad3-a414-82467f7188b3" containerName="main"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390168    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="86e98e6a-b1cf-4d08-b807-26397cd1b4a4" containerName="main"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390176    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0822059-3672-4595-967c-0796222fd966" containerName="payload-pre-processing"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390193    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc75808d-fb11-4820-939a-18037b0c2791" containerName="istio-proxy"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390199    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="e92bdf87-b7d6-4060-9615-70ae77214bc0" containerName="istio-proxy"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390204    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="778281fb-fe1b-4f27-bd43-62920826a0f9" containerName="istio-proxy"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390210    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dbd469d-d1f9-4a08-bdd8-6f4a29848e6b" containerName="payload-pre-processing"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390216    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="a11b8ed4-c24c-4919-959f-86cf8ecbacb2" containerName="istio-proxy"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390221    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bf2af8d-0d32-4191-ac7a-69f8aef47e1a" containerName="maas-api"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390226    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f728c20-eae2-4fd4-aa40-4094a75f5f16" containerName="maas-api"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390232    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ebbfb0-bf32-49c1-8feb-60b55eadbf55" containerName="maas-api"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390238    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddfb64f0-d446-48ec-90e7-6f2c1e36920f" containerName="istio-proxy"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390243    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a94e905-9533-4db9-b8d2-5c10538e0d0b" containerName="istio-proxy"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390250    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="2752235c-68d2-4be4-b1eb-d851d2b5c903" containerName="maas-api"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390255    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a94e905-9533-4db9-b8d2-5c10538e0d0b" containerName="istio-proxy"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390261    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b1105d-0395-4e80-a63b-7e8e0cce276d" containerName="istio-proxy"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390266    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac76439d-8c30-4810-b242-1a3e1ed2b556" containerName="maas-api"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390272    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd0afda-48df-44ec-bebd-52768b7bdbc0" containerName="istio-proxy"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390278    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd0afda-48df-44ec-bebd-52768b7bdbc0" containerName="istio-proxy"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390284    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a8525d-9951-4f19-b124-62e4fa91c6b9" containerName="main"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390289    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ab9bcb-f48a-404b-b787-9f8c43b58722" containerName="main"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390294    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="a87828aa-8038-4643-834a-bbdb78432874" containerName="main"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390299    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="55dbdf6b-c1f2-4e0a-907a-7cd59c30e989" containerName="main"
Oct 30 01:55:29.390721 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.390306    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="9de67abe-4f7c-441f-9e3e-cf34c74068d9" containerName="istio-proxy"
Oct 30 01:55:29.395216 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.395192    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.424541 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.424515    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"]
Oct 30 01:55:29.486556 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.486531    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-bc127f-openshift-default-dockercfg-q6vw7\""
Oct 30 01:55:29.586924 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.586893    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-workload-socket\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.587124 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.586937    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-podinfo\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.587124 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.586976    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-workload-certs\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.587124 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.587087    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-data\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.587290 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.587137    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-credential-socket\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.587290 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.587261    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5grfm\" (UniqueName: \"kubernetes.io/projected/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-kube-api-access-5grfm\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.587401 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.587320    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-token\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.587401 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.587364    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istiod-ca-cert\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.587512 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.587406    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-envoy\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.688925 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.688846    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5grfm\" (UniqueName: \"kubernetes.io/projected/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-kube-api-access-5grfm\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.688925 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.688890    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-token\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.688925 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.688917    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istiod-ca-cert\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.689272 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.688946    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-envoy\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.689272 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.688973    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-workload-socket\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.689272 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.689011    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-podinfo\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.689272 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.689085    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-workload-certs\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.689272 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.689141    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-data\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.689272 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.689203    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-credential-socket\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.689609 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.689537    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-workload-socket\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.689609 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.689586    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-workload-certs\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.689715 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.689621    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-credential-socket\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.689715 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.689649    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-data\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.689793 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.689772    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istiod-ca-cert\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.691335 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.691315    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-envoy\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.691435 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.691419    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-podinfo\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.714695 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.714665    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-token\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.714965 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.714936    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5grfm\" (UniqueName: \"kubernetes.io/projected/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-kube-api-access-5grfm\") pod \"e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") " pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:29.795779 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.794198    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86e98e6a-b1cf-4d08-b807-26397cd1b4a4" path="/var/lib/kubelet/pods/86e98e6a-b1cf-4d08-b807-26397cd1b4a4/volumes"
Oct 30 01:55:29.795779 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:29.794888    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0ab9bcb-f48a-404b-b787-9f8c43b58722" path="/var/lib/kubelet/pods/e0ab9bcb-f48a-404b-b787-9f8c43b58722/volumes"
Oct 30 01:55:30.012769 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:30.012737    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:30.327656 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:30.327628    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"]
Oct 30 01:55:30.329901 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:55:30.329865    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2b2d7c_c516_4da4_86c1_e958d87e0fb1.slice/crio-2afa4cb05da8387d152baa8fb252608baec89a624243ad4ed738ae7ba147a4c7 WatchSource:0}: Error finding container 2afa4cb05da8387d152baa8fb252608baec89a624243ad4ed738ae7ba147a4c7: Status 404 returned error can't find the container with id 2afa4cb05da8387d152baa8fb252608baec89a624243ad4ed738ae7ba147a4c7
Oct 30 01:55:30.332356 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:30.332324    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:55:30.332438 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:30.332389    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:55:30.332438 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:30.332421    4864 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 01:55:30.421790 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:30.421756    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4" event={"ID":"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1","Type":"ContainerStarted","Data":"92195df5300aecb23e72f062aeccd9cf3cef27151b083583adc960a6098c560e"}
Oct 30 01:55:30.421924 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:30.421807    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4" event={"ID":"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1","Type":"ContainerStarted","Data":"2afa4cb05da8387d152baa8fb252608baec89a624243ad4ed738ae7ba147a4c7"}
Oct 30 01:55:30.464352 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:30.464098    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4" podStartSLOduration=1.464060295 podStartE2EDuration="1.464060295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:30.463022562 +0000 UTC m=+4442.801124472" watchObservedRunningTime="2026-10-30 01:55:30.464060295 +0000 UTC m=+4442.802162206"
Oct 30 01:55:31.013239 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:31.013203    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:31.014625 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:31.014592    4864 patch_prober.go:29] interesting pod/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 01:55:31.014771 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:31.014649    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4" podUID="4d2b2d7c-c516-4da4-86c1-e958d87e0fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 01:55:31.800313 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:31.800283    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"]
Oct 30 01:55:32.013667 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:32.013630    4864 patch_prober.go:29] interesting pod/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 01:55:32.013841 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:32.013696    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4" podUID="4d2b2d7c-c516-4da4-86c1-e958d87e0fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 01:55:33.013670 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:33.013635    4864 patch_prober.go:29] interesting pod/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 01:55:33.014097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:33.013690    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4" podUID="4d2b2d7c-c516-4da4-86c1-e958d87e0fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 01:55:34.013201 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:34.013165    4864 patch_prober.go:29] interesting pod/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 01:55:34.013383 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:34.013216    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4" podUID="4d2b2d7c-c516-4da4-86c1-e958d87e0fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 01:55:35.013235 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:35.013202    4864 patch_prober.go:29] interesting pod/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 01:55:35.013639 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:35.013281    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4" podUID="4d2b2d7c-c516-4da4-86c1-e958d87e0fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 01:55:36.013041 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:36.013007    4864 patch_prober.go:29] interesting pod/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 01:55:36.013250 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:36.013060    4864 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4" podUID="4d2b2d7c-c516-4da4-86c1-e958d87e0fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 01:55:37.018468 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:37.018437    4864 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:37.018936 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:37.018848    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:37.019490 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:37.019468    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:55:42.606767 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:42.606661    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz"]
Oct 30 01:55:42.607572 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:42.607520    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz" podUID="53ecb5dd-2b7f-4139-8463-06583186f2e0" containerName="maas-api" containerID="cri-o://e74ff560b30cb50706c437dcd5e171fab898e52fc4c2a288e043731c08318f35" gracePeriod=30
Oct 30 01:55:42.872098 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:42.872015    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz"
Oct 30 01:55:42.921127 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:42.921094    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53ecb5dd-2b7f-4139-8463-06583186f2e0-kube-api-access-6l8gh\" (UniqueName: \"kubernetes.io/projected/53ecb5dd-2b7f-4139-8463-06583186f2e0-kube-api-access-6l8gh\") pod \"53ecb5dd-2b7f-4139-8463-06583186f2e0\" (UID: \"53ecb5dd-2b7f-4139-8463-06583186f2e0\") "
Oct 30 01:55:42.921308 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:42.921209    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53ecb5dd-2b7f-4139-8463-06583186f2e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53ecb5dd-2b7f-4139-8463-06583186f2e0-maas-api-tls\") pod \"53ecb5dd-2b7f-4139-8463-06583186f2e0\" (UID: \"53ecb5dd-2b7f-4139-8463-06583186f2e0\") "
Oct 30 01:55:42.921360 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:42.921317    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53ecb5dd-2b7f-4139-8463-06583186f2e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ecb5dd-2b7f-4139-8463-06583186f2e0-openshift-service-ca\") pod \"53ecb5dd-2b7f-4139-8463-06583186f2e0\" (UID: \"53ecb5dd-2b7f-4139-8463-06583186f2e0\") "
Oct 30 01:55:42.921395 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:42.921357    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53ecb5dd-2b7f-4139-8463-06583186f2e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ecb5dd-2b7f-4139-8463-06583186f2e0-metrics-certs\") pod \"53ecb5dd-2b7f-4139-8463-06583186f2e0\" (UID: \"53ecb5dd-2b7f-4139-8463-06583186f2e0\") "
Oct 30 01:55:42.921947 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:42.921891    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ecb5dd-2b7f-4139-8463-06583186f2e0-openshift-service-ca" pod "53ecb5dd-2b7f-4139-8463-06583186f2e0" (UID: "53ecb5dd-2b7f-4139-8463-06583186f2e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:42.923938 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:42.923909    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53ecb5dd-2b7f-4139-8463-06583186f2e0-maas-api-tls" pod "53ecb5dd-2b7f-4139-8463-06583186f2e0" (UID: "53ecb5dd-2b7f-4139-8463-06583186f2e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:42.924064 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:42.924031    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53ecb5dd-2b7f-4139-8463-06583186f2e0-metrics-certs" pod "53ecb5dd-2b7f-4139-8463-06583186f2e0" (UID: "53ecb5dd-2b7f-4139-8463-06583186f2e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:42.933234 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:42.933204    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ecb5dd-2b7f-4139-8463-06583186f2e0-kube-api-access-6l8gh" pod "53ecb5dd-2b7f-4139-8463-06583186f2e0" (UID: "53ecb5dd-2b7f-4139-8463-06583186f2e0"). InnerVolumeSpecName "kube-api-access-6l8gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:42.983042 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:42.983002    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz"]
Oct 30 01:55:42.983275 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:42.983249    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz" podUID="19b52d59-2121-4a46-b9a4-8c80742be6c3" containerName="maas-api" containerID="cri-o://313754e899fd358e7b5f6b067075a4bfec89221156ca30ac0dec298d2e13a248" gracePeriod=30
Oct 30 01:55:43.023099 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.023049    4864 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ecb5dd-2b7f-4139-8463-06583186f2e0-openshift-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:43.023099 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.023098    4864 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ecb5dd-2b7f-4139-8463-06583186f2e0-metrics-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:43.023301 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.023113    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l8gh\" (UniqueName: \"kubernetes.io/projected/53ecb5dd-2b7f-4139-8463-06583186f2e0-kube-api-access-6l8gh\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:43.023301 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.023128    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53ecb5dd-2b7f-4139-8463-06583186f2e0-maas-api-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:43.229064 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.229040    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz"
Oct 30 01:55:43.326354 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.326318    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19b52d59-2121-4a46-b9a4-8c80742be6c3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/19b52d59-2121-4a46-b9a4-8c80742be6c3-metrics-certs\") pod \"19b52d59-2121-4a46-b9a4-8c80742be6c3\" (UID: \"19b52d59-2121-4a46-b9a4-8c80742be6c3\") "
Oct 30 01:55:43.326550 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.326463    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19b52d59-2121-4a46-b9a4-8c80742be6c3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b52d59-2121-4a46-b9a4-8c80742be6c3-openshift-service-ca\") pod \"19b52d59-2121-4a46-b9a4-8c80742be6c3\" (UID: \"19b52d59-2121-4a46-b9a4-8c80742be6c3\") "
Oct 30 01:55:43.326550 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.326528    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b52d59-2121-4a46-b9a4-8c80742be6c3-kube-api-access-t9gqg\" (UniqueName: \"kubernetes.io/projected/19b52d59-2121-4a46-b9a4-8c80742be6c3-kube-api-access-t9gqg\") pod \"19b52d59-2121-4a46-b9a4-8c80742be6c3\" (UID: \"19b52d59-2121-4a46-b9a4-8c80742be6c3\") "
Oct 30 01:55:43.326683 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.326629    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19b52d59-2121-4a46-b9a4-8c80742be6c3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19b52d59-2121-4a46-b9a4-8c80742be6c3-maas-api-tls\") pod \"19b52d59-2121-4a46-b9a4-8c80742be6c3\" (UID: \"19b52d59-2121-4a46-b9a4-8c80742be6c3\") "
Oct 30 01:55:43.326877 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.326843    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b52d59-2121-4a46-b9a4-8c80742be6c3-openshift-service-ca" pod "19b52d59-2121-4a46-b9a4-8c80742be6c3" (UID: "19b52d59-2121-4a46-b9a4-8c80742be6c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:43.327024 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.326981    4864 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b52d59-2121-4a46-b9a4-8c80742be6c3-openshift-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:43.328626 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.328603    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19b52d59-2121-4a46-b9a4-8c80742be6c3-metrics-certs" pod "19b52d59-2121-4a46-b9a4-8c80742be6c3" (UID: "19b52d59-2121-4a46-b9a4-8c80742be6c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:43.328626 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.328616    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b52d59-2121-4a46-b9a4-8c80742be6c3-kube-api-access-t9gqg" pod "19b52d59-2121-4a46-b9a4-8c80742be6c3" (UID: "19b52d59-2121-4a46-b9a4-8c80742be6c3"). InnerVolumeSpecName "kube-api-access-t9gqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:43.328755 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.328637    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19b52d59-2121-4a46-b9a4-8c80742be6c3-maas-api-tls" pod "19b52d59-2121-4a46-b9a4-8c80742be6c3" (UID: "19b52d59-2121-4a46-b9a4-8c80742be6c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:43.428412 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.428328    4864 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19b52d59-2121-4a46-b9a4-8c80742be6c3-metrics-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:43.428412 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.428359    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9gqg\" (UniqueName: \"kubernetes.io/projected/19b52d59-2121-4a46-b9a4-8c80742be6c3-kube-api-access-t9gqg\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:43.428412 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.428370    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19b52d59-2121-4a46-b9a4-8c80742be6c3-maas-api-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:43.492157 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.492121    4864 generic.go:356] "Generic (PLEG): container finished" podID="53ecb5dd-2b7f-4139-8463-06583186f2e0" containerID="e74ff560b30cb50706c437dcd5e171fab898e52fc4c2a288e043731c08318f35" exitCode=0
Oct 30 01:55:43.492331 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.492165    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz"
Oct 30 01:55:43.492331 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.492208    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz" event={"ID":"53ecb5dd-2b7f-4139-8463-06583186f2e0","Type":"ContainerDied","Data":"e74ff560b30cb50706c437dcd5e171fab898e52fc4c2a288e043731c08318f35"}
Oct 30 01:55:43.492331 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.492257    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz" event={"ID":"53ecb5dd-2b7f-4139-8463-06583186f2e0","Type":"ContainerDied","Data":"80b38e3420063ff73e7c00f7ebef2ec0eacc5574a8891745436b628642ab8368"}
Oct 30 01:55:43.492331 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.492284    4864 scope.go:122] "RemoveContainer" containerID="e74ff560b30cb50706c437dcd5e171fab898e52fc4c2a288e043731c08318f35"
Oct 30 01:55:43.493597 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.493564    4864 generic.go:356] "Generic (PLEG): container finished" podID="19b52d59-2121-4a46-b9a4-8c80742be6c3" containerID="313754e899fd358e7b5f6b067075a4bfec89221156ca30ac0dec298d2e13a248" exitCode=0
Oct 30 01:55:43.493703 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.493618    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz" event={"ID":"19b52d59-2121-4a46-b9a4-8c80742be6c3","Type":"ContainerDied","Data":"313754e899fd358e7b5f6b067075a4bfec89221156ca30ac0dec298d2e13a248"}
Oct 30 01:55:43.493703 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.493655    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz" event={"ID":"19b52d59-2121-4a46-b9a4-8c80742be6c3","Type":"ContainerDied","Data":"78244041ced506bcdf4b16573e3c078dc565c9a90ad6ed2583cf195d86ec1af0"}
Oct 30 01:55:43.493703 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.493620    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz"
Oct 30 01:55:43.502988 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.502971    4864 scope.go:122] "RemoveContainer" containerID="e74ff560b30cb50706c437dcd5e171fab898e52fc4c2a288e043731c08318f35"
Oct 30 01:55:43.503309 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:55:43.503283    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e74ff560b30cb50706c437dcd5e171fab898e52fc4c2a288e043731c08318f35\": container with ID starting with e74ff560b30cb50706c437dcd5e171fab898e52fc4c2a288e043731c08318f35 not found: ID does not exist" containerID="e74ff560b30cb50706c437dcd5e171fab898e52fc4c2a288e043731c08318f35"
Oct 30 01:55:43.503446 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.503317    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e74ff560b30cb50706c437dcd5e171fab898e52fc4c2a288e043731c08318f35"} err="failed to get container status \"e74ff560b30cb50706c437dcd5e171fab898e52fc4c2a288e043731c08318f35\": rpc error: code = NotFound desc = could not find container \"e74ff560b30cb50706c437dcd5e171fab898e52fc4c2a288e043731c08318f35\": container with ID starting with e74ff560b30cb50706c437dcd5e171fab898e52fc4c2a288e043731c08318f35 not found: ID does not exist"
Oct 30 01:55:43.503446 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.503340    4864 scope.go:122] "RemoveContainer" containerID="313754e899fd358e7b5f6b067075a4bfec89221156ca30ac0dec298d2e13a248"
Oct 30 01:55:43.512746 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.512728    4864 scope.go:122] "RemoveContainer" containerID="313754e899fd358e7b5f6b067075a4bfec89221156ca30ac0dec298d2e13a248"
Oct 30 01:55:43.512996 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:55:43.512975    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313754e899fd358e7b5f6b067075a4bfec89221156ca30ac0dec298d2e13a248\": container with ID starting with 313754e899fd358e7b5f6b067075a4bfec89221156ca30ac0dec298d2e13a248 not found: ID does not exist" containerID="313754e899fd358e7b5f6b067075a4bfec89221156ca30ac0dec298d2e13a248"
Oct 30 01:55:43.513048 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.513002    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313754e899fd358e7b5f6b067075a4bfec89221156ca30ac0dec298d2e13a248"} err="failed to get container status \"313754e899fd358e7b5f6b067075a4bfec89221156ca30ac0dec298d2e13a248\": rpc error: code = NotFound desc = could not find container \"313754e899fd358e7b5f6b067075a4bfec89221156ca30ac0dec298d2e13a248\": container with ID starting with 313754e899fd358e7b5f6b067075a4bfec89221156ca30ac0dec298d2e13a248 not found: ID does not exist"
Oct 30 01:55:43.552001 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.551974    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz"]
Oct 30 01:55:43.556543 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.556519    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-664c2b-7498bc6765-pfhgz"]
Oct 30 01:55:43.587670 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.587640    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz"]
Oct 30 01:55:43.593081 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.593046    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-72ed13-5bf8448755-kf4xz"]
Oct 30 01:55:43.792789 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.792751    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19b52d59-2121-4a46-b9a4-8c80742be6c3" path="/var/lib/kubelet/pods/19b52d59-2121-4a46-b9a4-8c80742be6c3/volumes"
Oct 30 01:55:43.793440 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:43.793418    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53ecb5dd-2b7f-4139-8463-06583186f2e0" path="/var/lib/kubelet/pods/53ecb5dd-2b7f-4139-8463-06583186f2e0/volumes"
Oct 30 01:55:44.152959 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:44.152873    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx"]
Oct 30 01:55:44.190078 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:44.190048    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq"]
Oct 30 01:55:46.820647 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:46.820588    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz" podUID="e8667b92-fdd3-49f8-bd1b-541d378530ef" containerName="main" containerID="cri-o://30d8cef50595ded612d2e1625e9428f09800de4c7993e29666d4f18aa3473529" gracePeriod=45
Oct 30 01:55:47.087384 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.087332    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:55:47.168189 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.168154    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8667b92-fdd3-49f8-bd1b-541d378530ef-kube-api-access-7ckwt\" (UniqueName: \"kubernetes.io/projected/e8667b92-fdd3-49f8-bd1b-541d378530ef-kube-api-access-7ckwt\") pod \"e8667b92-fdd3-49f8-bd1b-541d378530ef\" (UID: \"e8667b92-fdd3-49f8-bd1b-541d378530ef\") "
Oct 30 01:55:47.168350 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.168203    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-dshm\") pod \"e8667b92-fdd3-49f8-bd1b-541d378530ef\" (UID: \"e8667b92-fdd3-49f8-bd1b-541d378530ef\") "
Oct 30 01:55:47.168350 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.168247    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-tmp-dir\") pod \"e8667b92-fdd3-49f8-bd1b-541d378530ef\" (UID: \"e8667b92-fdd3-49f8-bd1b-541d378530ef\") "
Oct 30 01:55:47.168350 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.168281    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-model-cache\") pod \"e8667b92-fdd3-49f8-bd1b-541d378530ef\" (UID: \"e8667b92-fdd3-49f8-bd1b-541d378530ef\") "
Oct 30 01:55:47.168350 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.168333    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8667b92-fdd3-49f8-bd1b-541d378530ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/e8667b92-fdd3-49f8-bd1b-541d378530ef-tls-certs\") pod \"e8667b92-fdd3-49f8-bd1b-541d378530ef\" (UID: \"e8667b92-fdd3-49f8-bd1b-541d378530ef\") "
Oct 30 01:55:47.168545 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.168398    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-home\" (UniqueName: \"kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-home\") pod \"e8667b92-fdd3-49f8-bd1b-541d378530ef\" (UID: \"e8667b92-fdd3-49f8-bd1b-541d378530ef\") "
Oct 30 01:55:47.168616 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.168579    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-model-cache" pod "e8667b92-fdd3-49f8-bd1b-541d378530ef" (UID: "e8667b92-fdd3-49f8-bd1b-541d378530ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:47.168682 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.168610    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-tmp-dir" pod "e8667b92-fdd3-49f8-bd1b-541d378530ef" (UID: "e8667b92-fdd3-49f8-bd1b-541d378530ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:47.168764 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.168742    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-home" pod "e8667b92-fdd3-49f8-bd1b-541d378530ef" (UID: "e8667b92-fdd3-49f8-bd1b-541d378530ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:47.168946 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.168927    4864 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-model-cache\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:47.169000 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.168952    4864 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-home\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:47.169000 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.168965    4864 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-tmp-dir\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:47.170519 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.170492    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8667b92-fdd3-49f8-bd1b-541d378530ef-tls-certs" pod "e8667b92-fdd3-49f8-bd1b-541d378530ef" (UID: "e8667b92-fdd3-49f8-bd1b-541d378530ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:47.170822 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.170803    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-dshm" pod "e8667b92-fdd3-49f8-bd1b-541d378530ef" (UID: "e8667b92-fdd3-49f8-bd1b-541d378530ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:47.170892 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.170809    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8667b92-fdd3-49f8-bd1b-541d378530ef-kube-api-access-7ckwt" pod "e8667b92-fdd3-49f8-bd1b-541d378530ef" (UID: "e8667b92-fdd3-49f8-bd1b-541d378530ef"). InnerVolumeSpecName "kube-api-access-7ckwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:47.270236 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.270201    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ckwt\" (UniqueName: \"kubernetes.io/projected/e8667b92-fdd3-49f8-bd1b-541d378530ef-kube-api-access-7ckwt\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:47.270236 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.270230    4864 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8667b92-fdd3-49f8-bd1b-541d378530ef-dshm\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:47.270236 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.270243    4864 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8667b92-fdd3-49f8-bd1b-541d378530ef-tls-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:47.518497 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.518460    4864 generic.go:356] "Generic (PLEG): container finished" podID="e8667b92-fdd3-49f8-bd1b-541d378530ef" containerID="30d8cef50595ded612d2e1625e9428f09800de4c7993e29666d4f18aa3473529" exitCode=0
Oct 30 01:55:47.518497 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.518501    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"
Oct 30 01:55:47.518775 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.518543    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz" event={"ID":"e8667b92-fdd3-49f8-bd1b-541d378530ef","Type":"ContainerDied","Data":"30d8cef50595ded612d2e1625e9428f09800de4c7993e29666d4f18aa3473529"}
Oct 30 01:55:47.518775 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.518582    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz" event={"ID":"e8667b92-fdd3-49f8-bd1b-541d378530ef","Type":"ContainerDied","Data":"8585aa526ecefb1baea866d03a35a14be32381c913f8f736fa87b13feded584e"}
Oct 30 01:55:47.518775 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.518600    4864 scope.go:122] "RemoveContainer" containerID="30d8cef50595ded612d2e1625e9428f09800de4c7993e29666d4f18aa3473529"
Oct 30 01:55:47.528644 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.528615    4864 scope.go:122] "RemoveContainer" containerID="30d8cef50595ded612d2e1625e9428f09800de4c7993e29666d4f18aa3473529"
Oct 30 01:55:47.528918 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:55:47.528893    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30d8cef50595ded612d2e1625e9428f09800de4c7993e29666d4f18aa3473529\": container with ID starting with 30d8cef50595ded612d2e1625e9428f09800de4c7993e29666d4f18aa3473529 not found: ID does not exist" containerID="30d8cef50595ded612d2e1625e9428f09800de4c7993e29666d4f18aa3473529"
Oct 30 01:55:47.528994 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.528926    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30d8cef50595ded612d2e1625e9428f09800de4c7993e29666d4f18aa3473529"} err="failed to get container status \"30d8cef50595ded612d2e1625e9428f09800de4c7993e29666d4f18aa3473529\": rpc error: code = NotFound desc = could not find container \"30d8cef50595ded612d2e1625e9428f09800de4c7993e29666d4f18aa3473529\": container with ID starting with 30d8cef50595ded612d2e1625e9428f09800de4c7993e29666d4f18aa3473529 not found: ID does not exist"
Oct 30 01:55:47.564818 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.564789    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"]
Oct 30 01:55:47.570615 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.570584    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-bc127f/rate-test-model-bc127f-kserve-5fd695c845-g8ppz"]
Oct 30 01:55:47.792131 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:47.792034    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8667b92-fdd3-49f8-bd1b-541d378530ef" path="/var/lib/kubelet/pods/e8667b92-fdd3-49f8-bd1b-541d378530ef/volumes"
Oct 30 01:55:49.039273 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:55:49.039239    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:55:49.153985 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:49.153925    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx" podUID="83de80bb-1c42-4450-9019-abc0e614280b" containerName="payload-processing" containerID="cri-o://253efa8f059d061db3bcd570c42249fd7beeb1ef8047262cf797fa5eee3163d8" gracePeriod=25
Oct 30 01:55:49.191263 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:49.191217    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq" podUID="46b674ae-7af3-4ef9-a04a-bea2b1071c1d" containerName="payload-pre-processing" containerID="cri-o://e991e8f9be961ecf8bb38224e4ed311bb6e308b0e082400b3cfe4f77593ed895" gracePeriod=25
Oct 30 01:55:53.524342 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:53.524306    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"]
Oct 30 01:55:53.524947 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:53.524578    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz" podUID="eab47e5b-e868-483e-92a2-e3414cc1dda2" containerName="istio-proxy" containerID="cri-o://1e4c6877bd214bf6f466de8c25bde514f13f7614ad44dca3959e3b40fb12950a" gracePeriod=30
Oct 30 01:55:57.420264 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:57.420226    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"]
Oct 30 01:55:57.420728 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:57.420504    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8" podUID="41fd0656-df2b-4cd7-9f6a-ac93a18f9579" containerName="istio-proxy" containerID="cri-o://173398fe52fbe34f9a33bbf6b10703c6c903b704cea5369ff43d6c2094c96645" gracePeriod=30
Oct 30 01:55:57.530234 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:57.530189    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx" podUID="83de80bb-1c42-4450-9019-abc0e614280b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:55:57.530526 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:57.530498    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq" podUID="46b674ae-7af3-4ef9-a04a-bea2b1071c1d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:55:58.781467 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.781441    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:55:58.895207 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.895129    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-credential-socket\") pod \"eab47e5b-e868-483e-92a2-e3414cc1dda2\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") "
Oct 30 01:55:58.895207 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.895177    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-workload-socket\") pod \"eab47e5b-e868-483e-92a2-e3414cc1dda2\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") "
Oct 30 01:55:58.895420 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.895218    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eab47e5b-e868-483e-92a2-e3414cc1dda2-kube-api-access-kbzs2\" (UniqueName: \"kubernetes.io/projected/eab47e5b-e868-483e-92a2-e3414cc1dda2-kube-api-access-kbzs2\") pod \"eab47e5b-e868-483e-92a2-e3414cc1dda2\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") "
Oct 30 01:55:58.895420 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.895252    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-envoy\") pod \"eab47e5b-e868-483e-92a2-e3414cc1dda2\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") "
Oct 30 01:55:58.895420 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.895317    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-podinfo\") pod \"eab47e5b-e868-483e-92a2-e3414cc1dda2\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") "
Oct 30 01:55:58.895420 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.895342    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-data\") pod \"eab47e5b-e868-483e-92a2-e3414cc1dda2\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") "
Oct 30 01:55:58.895420 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.895372    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-workload-certs\") pod \"eab47e5b-e868-483e-92a2-e3414cc1dda2\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") "
Oct 30 01:55:58.895700 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.895443    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-token\" (UniqueName: \"kubernetes.io/projected/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-token\") pod \"eab47e5b-e868-483e-92a2-e3414cc1dda2\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") "
Oct 30 01:55:58.895700 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.895449    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-credential-socket" pod "eab47e5b-e868-483e-92a2-e3414cc1dda2" (UID: "eab47e5b-e868-483e-92a2-e3414cc1dda2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:58.895700 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.895520    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eab47e5b-e868-483e-92a2-e3414cc1dda2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab47e5b-e868-483e-92a2-e3414cc1dda2-istiod-ca-cert\") pod \"eab47e5b-e868-483e-92a2-e3414cc1dda2\" (UID: \"eab47e5b-e868-483e-92a2-e3414cc1dda2\") "
Oct 30 01:55:58.895700 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.895533    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-workload-socket" pod "eab47e5b-e868-483e-92a2-e3414cc1dda2" (UID: "eab47e5b-e868-483e-92a2-e3414cc1dda2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:58.895700 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.895633    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-data" pod "eab47e5b-e868-483e-92a2-e3414cc1dda2" (UID: "eab47e5b-e868-483e-92a2-e3414cc1dda2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:58.895977 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.895736    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-workload-certs" pod "eab47e5b-e868-483e-92a2-e3414cc1dda2" (UID: "eab47e5b-e868-483e-92a2-e3414cc1dda2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:58.895977 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.895925    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eab47e5b-e868-483e-92a2-e3414cc1dda2-istiod-ca-cert" pod "eab47e5b-e868-483e-92a2-e3414cc1dda2" (UID: "eab47e5b-e868-483e-92a2-e3414cc1dda2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:58.896171 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.896109    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:58.896171 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.896130    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:58.896171 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.896145    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab47e5b-e868-483e-92a2-e3414cc1dda2-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:58.896171 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.896159    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:58.896420 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.896173    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:58.897649 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.897624    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-podinfo" pod "eab47e5b-e868-483e-92a2-e3414cc1dda2" (UID: "eab47e5b-e868-483e-92a2-e3414cc1dda2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:55:58.897649 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.897629    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eab47e5b-e868-483e-92a2-e3414cc1dda2-kube-api-access-kbzs2" pod "eab47e5b-e868-483e-92a2-e3414cc1dda2" (UID: "eab47e5b-e868-483e-92a2-e3414cc1dda2"). InnerVolumeSpecName "kube-api-access-kbzs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:58.897801 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.897667    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-token" pod "eab47e5b-e868-483e-92a2-e3414cc1dda2" (UID: "eab47e5b-e868-483e-92a2-e3414cc1dda2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:58.898119 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.898099    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-envoy" pod "eab47e5b-e868-483e-92a2-e3414cc1dda2" (UID: "eab47e5b-e868-483e-92a2-e3414cc1dda2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:58.997585 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.997554    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:58.997585 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.997582    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbzs2\" (UniqueName: \"kubernetes.io/projected/eab47e5b-e868-483e-92a2-e3414cc1dda2-kube-api-access-kbzs2\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:58.997585 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.997593    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:58.997907 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:58.997602    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab47e5b-e868-483e-92a2-e3414cc1dda2-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:55:59.584988 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:59.584953    4864 generic.go:356] "Generic (PLEG): container finished" podID="eab47e5b-e868-483e-92a2-e3414cc1dda2" containerID="1e4c6877bd214bf6f466de8c25bde514f13f7614ad44dca3959e3b40fb12950a" exitCode=0
Oct 30 01:55:59.585187 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:59.585001    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"
Oct 30 01:55:59.585187 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:59.585035    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz" event={"ID":"eab47e5b-e868-483e-92a2-e3414cc1dda2","Type":"ContainerDied","Data":"1e4c6877bd214bf6f466de8c25bde514f13f7614ad44dca3959e3b40fb12950a"}
Oct 30 01:55:59.585187 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:59.585105    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz" event={"ID":"eab47e5b-e868-483e-92a2-e3414cc1dda2","Type":"ContainerDied","Data":"38f6a833e62861e0e98820ec2290767679313d4bc9f4693ab17ca37364753753"}
Oct 30 01:55:59.585187 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:59.585128    4864 scope.go:122] "RemoveContainer" containerID="1e4c6877bd214bf6f466de8c25bde514f13f7614ad44dca3959e3b40fb12950a"
Oct 30 01:55:59.595305 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:59.595286    4864 scope.go:122] "RemoveContainer" containerID="1e4c6877bd214bf6f466de8c25bde514f13f7614ad44dca3959e3b40fb12950a"
Oct 30 01:55:59.595591 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:55:59.595573    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e4c6877bd214bf6f466de8c25bde514f13f7614ad44dca3959e3b40fb12950a\": container with ID starting with 1e4c6877bd214bf6f466de8c25bde514f13f7614ad44dca3959e3b40fb12950a not found: ID does not exist" containerID="1e4c6877bd214bf6f466de8c25bde514f13f7614ad44dca3959e3b40fb12950a"
Oct 30 01:55:59.595642 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:59.595600    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e4c6877bd214bf6f466de8c25bde514f13f7614ad44dca3959e3b40fb12950a"} err="failed to get container status \"1e4c6877bd214bf6f466de8c25bde514f13f7614ad44dca3959e3b40fb12950a\": rpc error: code = NotFound desc = could not find container \"1e4c6877bd214bf6f466de8c25bde514f13f7614ad44dca3959e3b40fb12950a\": container with ID starting with 1e4c6877bd214bf6f466de8c25bde514f13f7614ad44dca3959e3b40fb12950a not found: ID does not exist"
Oct 30 01:55:59.633645 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:59.633612    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"]
Oct 30 01:55:59.642697 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:59.642667    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-kg6tz"]
Oct 30 01:55:59.793276 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:55:59.793241    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eab47e5b-e868-483e-92a2-e3414cc1dda2" path="/var/lib/kubelet/pods/eab47e5b-e868-483e-92a2-e3414cc1dda2/volumes"
Oct 30 01:56:02.607412 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.607374    4864 generic.go:356] "Generic (PLEG): container finished" podID="41fd0656-df2b-4cd7-9f6a-ac93a18f9579" containerID="173398fe52fbe34f9a33bbf6b10703c6c903b704cea5369ff43d6c2094c96645" exitCode=0
Oct 30 01:56:02.608021 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.607487    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8" event={"ID":"41fd0656-df2b-4cd7-9f6a-ac93a18f9579","Type":"ContainerDied","Data":"173398fe52fbe34f9a33bbf6b10703c6c903b704cea5369ff43d6c2094c96645"}
Oct 30 01:56:02.608021 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.607581    4864 scope.go:122] "RemoveContainer" containerID="40a68cf616bf0a6e16e226ba8b6d9b729a35261b1f4d552168efcb17024f35bd"
Oct 30 01:56:02.674011 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.673990    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:56:02.837230 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.837195    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-credential-socket\") pod \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") "
Oct 30 01:56:02.837443 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.837246    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-kube-api-access-mltj6\" (UniqueName: \"kubernetes.io/projected/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-kube-api-access-mltj6\") pod \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") "
Oct 30 01:56:02.837443 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.837290    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-data\") pod \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") "
Oct 30 01:56:02.837443 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.837321    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istiod-ca-cert\") pod \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") "
Oct 30 01:56:02.837443 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.837343    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-envoy\") pod \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") "
Oct 30 01:56:02.837443 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.837422    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-workload-certs\") pod \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") "
Oct 30 01:56:02.837793 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.837451    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-token\" (UniqueName: \"kubernetes.io/projected/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-token\") pod \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") "
Oct 30 01:56:02.837793 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.837533    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-credential-socket" pod "41fd0656-df2b-4cd7-9f6a-ac93a18f9579" (UID: "41fd0656-df2b-4cd7-9f6a-ac93a18f9579"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:56:02.837793 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.837563    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-data" pod "41fd0656-df2b-4cd7-9f6a-ac93a18f9579" (UID: "41fd0656-df2b-4cd7-9f6a-ac93a18f9579"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:56:02.837793 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.837551    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-podinfo\") pod \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") "
Oct 30 01:56:02.837793 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.837681    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-workload-socket\") pod \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\" (UID: \"41fd0656-df2b-4cd7-9f6a-ac93a18f9579\") "
Oct 30 01:56:02.837793 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.837764    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istiod-ca-cert" pod "41fd0656-df2b-4cd7-9f6a-ac93a18f9579" (UID: "41fd0656-df2b-4cd7-9f6a-ac93a18f9579"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:56:02.838110 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.837818    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-workload-certs" pod "41fd0656-df2b-4cd7-9f6a-ac93a18f9579" (UID: "41fd0656-df2b-4cd7-9f6a-ac93a18f9579"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:56:02.838110 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.837907    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-workload-socket" pod "41fd0656-df2b-4cd7-9f6a-ac93a18f9579" (UID: "41fd0656-df2b-4cd7-9f6a-ac93a18f9579"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:56:02.838227 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.838132    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:56:02.838227 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.838158    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:56:02.838227 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.838174    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:56:02.838227 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.838189    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:56:02.838227 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.838203    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:56:02.839844 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.839816    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-kube-api-access-mltj6" pod "41fd0656-df2b-4cd7-9f6a-ac93a18f9579" (UID: "41fd0656-df2b-4cd7-9f6a-ac93a18f9579"). InnerVolumeSpecName "kube-api-access-mltj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:56:02.839844 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.839831    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-envoy" pod "41fd0656-df2b-4cd7-9f6a-ac93a18f9579" (UID: "41fd0656-df2b-4cd7-9f6a-ac93a18f9579"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:56:02.840002 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.839839    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-podinfo" pod "41fd0656-df2b-4cd7-9f6a-ac93a18f9579" (UID: "41fd0656-df2b-4cd7-9f6a-ac93a18f9579"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:56:02.840002 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.839896    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-token" pod "41fd0656-df2b-4cd7-9f6a-ac93a18f9579" (UID: "41fd0656-df2b-4cd7-9f6a-ac93a18f9579"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:56:02.939611 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.939575    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:56:02.939611 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.939607    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:56:02.939860 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.939620    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mltj6\" (UniqueName: \"kubernetes.io/projected/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-kube-api-access-mltj6\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:56:02.939860 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:02.939635    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fd0656-df2b-4cd7-9f6a-ac93a18f9579-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:56:03.613647 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:03.613615    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"
Oct 30 01:56:03.614122 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:03.613660    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8" event={"ID":"41fd0656-df2b-4cd7-9f6a-ac93a18f9579","Type":"ContainerDied","Data":"12cf0bb74e4a686b0eada052c34d8cedf6cdf4d3a7fe6ab00c2cc29facc71135"}
Oct 30 01:56:03.614122 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:03.613727    4864 scope.go:122] "RemoveContainer" containerID="173398fe52fbe34f9a33bbf6b10703c6c903b704cea5369ff43d6c2094c96645"
Oct 30 01:56:03.670265 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:03.670227    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"]
Oct 30 01:56:03.674036 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:03.674005    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-2f820e-openshift-default-5d9685957c-5g8v8"]
Oct 30 01:56:03.793280 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:03.793247    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41fd0656-df2b-4cd7-9f6a-ac93a18f9579" path="/var/lib/kubelet/pods/41fd0656-df2b-4cd7-9f6a-ac93a18f9579/volumes"
Oct 30 01:56:06.795412 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:06.795383    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s"]
Oct 30 01:56:06.795823 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:06.795585    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s" podUID="e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c" containerName="maas-api" containerID="cri-o://3d2cd75496e25b30666e8cbe951e60d725610a27657547993e87d2a3648ae4af" gracePeriod=30
Oct 30 01:56:07.050669 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.050611    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s"
Oct 30 01:56:07.180884 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.180853    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-kube-api-access-wwvqh\" (UniqueName: \"kubernetes.io/projected/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-kube-api-access-wwvqh\") pod \"e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c\" (UID: \"e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c\") "
Oct 30 01:56:07.180884 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.180891    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-metrics-certs\") pod \"e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c\" (UID: \"e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c\") "
Oct 30 01:56:07.181147 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.180925    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-maas-api-tls\") pod \"e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c\" (UID: \"e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c\") "
Oct 30 01:56:07.181147 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.180944    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-openshift-service-ca\") pod \"e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c\" (UID: \"e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c\") "
Oct 30 01:56:07.181433 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.181398    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-openshift-service-ca" pod "e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c" (UID: "e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:56:07.183097 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.183040    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-metrics-certs" pod "e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c" (UID: "e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:56:07.183220 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.183126    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-kube-api-access-wwvqh" pod "e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c" (UID: "e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c"). InnerVolumeSpecName "kube-api-access-wwvqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:56:07.183220 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.183146    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-maas-api-tls" pod "e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c" (UID: "e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:56:07.282802 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.282763    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwvqh\" (UniqueName: \"kubernetes.io/projected/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-kube-api-access-wwvqh\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:56:07.282802 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.282793    4864 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-metrics-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:56:07.282802 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.282804    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-maas-api-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:56:07.283034 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.282812    4864 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c-openshift-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:56:07.529600 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.529565    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq" podUID="46b674ae-7af3-4ef9-a04a-bea2b1071c1d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:56:07.529600 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.529587    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx" podUID="83de80bb-1c42-4450-9019-abc0e614280b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:56:07.637294 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.637258    4864 generic.go:356] "Generic (PLEG): container finished" podID="e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c" containerID="3d2cd75496e25b30666e8cbe951e60d725610a27657547993e87d2a3648ae4af" exitCode=0
Oct 30 01:56:07.637468 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.637307    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s"
Oct 30 01:56:07.637468 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.637355    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s" event={"ID":"e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c","Type":"ContainerDied","Data":"3d2cd75496e25b30666e8cbe951e60d725610a27657547993e87d2a3648ae4af"}
Oct 30 01:56:07.637468 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.637407    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s" event={"ID":"e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c","Type":"ContainerDied","Data":"43b39e9e6e481a9198c46eafebd18fdef183469984518b693db53fc86cb2213c"}
Oct 30 01:56:07.637468 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.637430    4864 scope.go:122] "RemoveContainer" containerID="3d2cd75496e25b30666e8cbe951e60d725610a27657547993e87d2a3648ae4af"
Oct 30 01:56:07.647400 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.647380    4864 scope.go:122] "RemoveContainer" containerID="3d2cd75496e25b30666e8cbe951e60d725610a27657547993e87d2a3648ae4af"
Oct 30 01:56:07.647667 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:56:07.647650    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d2cd75496e25b30666e8cbe951e60d725610a27657547993e87d2a3648ae4af\": container with ID starting with 3d2cd75496e25b30666e8cbe951e60d725610a27657547993e87d2a3648ae4af not found: ID does not exist" containerID="3d2cd75496e25b30666e8cbe951e60d725610a27657547993e87d2a3648ae4af"
Oct 30 01:56:07.647710 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.647676    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d2cd75496e25b30666e8cbe951e60d725610a27657547993e87d2a3648ae4af"} err="failed to get container status \"3d2cd75496e25b30666e8cbe951e60d725610a27657547993e87d2a3648ae4af\": rpc error: code = NotFound desc = could not find container \"3d2cd75496e25b30666e8cbe951e60d725610a27657547993e87d2a3648ae4af\": container with ID starting with 3d2cd75496e25b30666e8cbe951e60d725610a27657547993e87d2a3648ae4af not found: ID does not exist"
Oct 30 01:56:07.684322 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.684283    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s"]
Oct 30 01:56:07.688620 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.688594    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9bfe36-77b59b4b9d-dlf2s"]
Oct 30 01:56:07.795127 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:07.795044    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c" path="/var/lib/kubelet/pods/e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c/volumes"
Oct 30 01:56:09.467527 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.467497    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx"
Oct 30 01:56:09.477185 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.477163    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq"
Oct 30 01:56:09.505188 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.505161    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46b674ae-7af3-4ef9-a04a-bea2b1071c1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b674ae-7af3-4ef9-a04a-bea2b1071c1d-plugins-config-volume\") pod \"46b674ae-7af3-4ef9-a04a-bea2b1071c1d\" (UID: \"46b674ae-7af3-4ef9-a04a-bea2b1071c1d\") "
Oct 30 01:56:09.505359 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.505267    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83de80bb-1c42-4450-9019-abc0e614280b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83de80bb-1c42-4450-9019-abc0e614280b-plugins-config-volume\") pod \"83de80bb-1c42-4450-9019-abc0e614280b\" (UID: \"83de80bb-1c42-4450-9019-abc0e614280b\") "
Oct 30 01:56:09.505426 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.505369    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b674ae-7af3-4ef9-a04a-bea2b1071c1d-kube-api-access-txwnc\" (UniqueName: \"kubernetes.io/projected/46b674ae-7af3-4ef9-a04a-bea2b1071c1d-kube-api-access-txwnc\") pod \"46b674ae-7af3-4ef9-a04a-bea2b1071c1d\" (UID: \"46b674ae-7af3-4ef9-a04a-bea2b1071c1d\") "
Oct 30 01:56:09.505426 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.505402    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83de80bb-1c42-4450-9019-abc0e614280b-kube-api-access-4dncx\" (UniqueName: \"kubernetes.io/projected/83de80bb-1c42-4450-9019-abc0e614280b-kube-api-access-4dncx\") pod \"83de80bb-1c42-4450-9019-abc0e614280b\" (UID: \"83de80bb-1c42-4450-9019-abc0e614280b\") "
Oct 30 01:56:09.505629 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.505602    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b674ae-7af3-4ef9-a04a-bea2b1071c1d-plugins-config-volume" pod "46b674ae-7af3-4ef9-a04a-bea2b1071c1d" (UID: "46b674ae-7af3-4ef9-a04a-bea2b1071c1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:56:09.505629 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.505635    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83de80bb-1c42-4450-9019-abc0e614280b-plugins-config-volume" pod "83de80bb-1c42-4450-9019-abc0e614280b" (UID: "83de80bb-1c42-4450-9019-abc0e614280b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:56:09.505835 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.505789    4864 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b674ae-7af3-4ef9-a04a-bea2b1071c1d-plugins-config-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:56:09.505835 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.505809    4864 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83de80bb-1c42-4450-9019-abc0e614280b-plugins-config-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:56:09.508523 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.508066    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83de80bb-1c42-4450-9019-abc0e614280b-kube-api-access-4dncx" pod "83de80bb-1c42-4450-9019-abc0e614280b" (UID: "83de80bb-1c42-4450-9019-abc0e614280b"). InnerVolumeSpecName "kube-api-access-4dncx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:56:09.508523 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.508132    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b674ae-7af3-4ef9-a04a-bea2b1071c1d-kube-api-access-txwnc" pod "46b674ae-7af3-4ef9-a04a-bea2b1071c1d" (UID: "46b674ae-7af3-4ef9-a04a-bea2b1071c1d"). InnerVolumeSpecName "kube-api-access-txwnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:56:09.606682 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.606647    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txwnc\" (UniqueName: \"kubernetes.io/projected/46b674ae-7af3-4ef9-a04a-bea2b1071c1d-kube-api-access-txwnc\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:56:09.606682 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.606679    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dncx\" (UniqueName: \"kubernetes.io/projected/83de80bb-1c42-4450-9019-abc0e614280b-kube-api-access-4dncx\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:56:09.651397 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.651363    4864 generic.go:356] "Generic (PLEG): container finished" podID="83de80bb-1c42-4450-9019-abc0e614280b" containerID="253efa8f059d061db3bcd570c42249fd7beeb1ef8047262cf797fa5eee3163d8" exitCode=0
Oct 30 01:56:09.651638 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.651415    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx"
Oct 30 01:56:09.651638 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.651452    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx" event={"ID":"83de80bb-1c42-4450-9019-abc0e614280b","Type":"ContainerDied","Data":"253efa8f059d061db3bcd570c42249fd7beeb1ef8047262cf797fa5eee3163d8"}
Oct 30 01:56:09.651638 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.651514    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx" event={"ID":"83de80bb-1c42-4450-9019-abc0e614280b","Type":"ContainerDied","Data":"739ec9c8585d9b7f9523908b651a7bc38db058d537bf81aae0ef800d9f05d940"}
Oct 30 01:56:09.651638 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.651548    4864 scope.go:122] "RemoveContainer" containerID="253efa8f059d061db3bcd570c42249fd7beeb1ef8047262cf797fa5eee3163d8"
Oct 30 01:56:09.652840 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.652816    4864 generic.go:356] "Generic (PLEG): container finished" podID="46b674ae-7af3-4ef9-a04a-bea2b1071c1d" containerID="e991e8f9be961ecf8bb38224e4ed311bb6e308b0e082400b3cfe4f77593ed895" exitCode=0
Oct 30 01:56:09.652941 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.652866    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq"
Oct 30 01:56:09.652941 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.652877    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq" event={"ID":"46b674ae-7af3-4ef9-a04a-bea2b1071c1d","Type":"ContainerDied","Data":"e991e8f9be961ecf8bb38224e4ed311bb6e308b0e082400b3cfe4f77593ed895"}
Oct 30 01:56:09.652941 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.652925    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq" event={"ID":"46b674ae-7af3-4ef9-a04a-bea2b1071c1d","Type":"ContainerDied","Data":"5f4c40dfe3510a24ac859996623da1180333905c1950226daca36df9ce4b0964"}
Oct 30 01:56:09.665976 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.665960    4864 scope.go:122] "RemoveContainer" containerID="253efa8f059d061db3bcd570c42249fd7beeb1ef8047262cf797fa5eee3163d8"
Oct 30 01:56:09.666295 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:56:09.666275    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253efa8f059d061db3bcd570c42249fd7beeb1ef8047262cf797fa5eee3163d8\": container with ID starting with 253efa8f059d061db3bcd570c42249fd7beeb1ef8047262cf797fa5eee3163d8 not found: ID does not exist" containerID="253efa8f059d061db3bcd570c42249fd7beeb1ef8047262cf797fa5eee3163d8"
Oct 30 01:56:09.666358 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.666302    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253efa8f059d061db3bcd570c42249fd7beeb1ef8047262cf797fa5eee3163d8"} err="failed to get container status \"253efa8f059d061db3bcd570c42249fd7beeb1ef8047262cf797fa5eee3163d8\": rpc error: code = NotFound desc = could not find container \"253efa8f059d061db3bcd570c42249fd7beeb1ef8047262cf797fa5eee3163d8\": container with ID starting with 253efa8f059d061db3bcd570c42249fd7beeb1ef8047262cf797fa5eee3163d8 not found: ID does not exist"
Oct 30 01:56:09.666358 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.666319    4864 scope.go:122] "RemoveContainer" containerID="e991e8f9be961ecf8bb38224e4ed311bb6e308b0e082400b3cfe4f77593ed895"
Oct 30 01:56:09.674375 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.674355    4864 scope.go:122] "RemoveContainer" containerID="e991e8f9be961ecf8bb38224e4ed311bb6e308b0e082400b3cfe4f77593ed895"
Oct 30 01:56:09.674621 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:56:09.674601    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e991e8f9be961ecf8bb38224e4ed311bb6e308b0e082400b3cfe4f77593ed895\": container with ID starting with e991e8f9be961ecf8bb38224e4ed311bb6e308b0e082400b3cfe4f77593ed895 not found: ID does not exist" containerID="e991e8f9be961ecf8bb38224e4ed311bb6e308b0e082400b3cfe4f77593ed895"
Oct 30 01:56:09.674692 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.674633    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e991e8f9be961ecf8bb38224e4ed311bb6e308b0e082400b3cfe4f77593ed895"} err="failed to get container status \"e991e8f9be961ecf8bb38224e4ed311bb6e308b0e082400b3cfe4f77593ed895\": rpc error: code = NotFound desc = could not find container \"e991e8f9be961ecf8bb38224e4ed311bb6e308b0e082400b3cfe4f77593ed895\": container with ID starting with e991e8f9be961ecf8bb38224e4ed311bb6e308b0e082400b3cfe4f77593ed895 not found: ID does not exist"
Oct 30 01:56:09.709032 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.709007    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx"]
Oct 30 01:56:09.714487 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.714460    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9bfe36-77d9f54ddd-z9msx"]
Oct 30 01:56:09.744822 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.744798    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq"]
Oct 30 01:56:09.749194 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.749173    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9bfe36-68745455d7-9s4sq"]
Oct 30 01:56:09.792393 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.792366    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46b674ae-7af3-4ef9-a04a-bea2b1071c1d" path="/var/lib/kubelet/pods/46b674ae-7af3-4ef9-a04a-bea2b1071c1d/volumes"
Oct 30 01:56:09.792750 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:09.792738    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83de80bb-1c42-4450-9019-abc0e614280b" path="/var/lib/kubelet/pods/83de80bb-1c42-4450-9019-abc0e614280b/volumes"
Oct 30 01:56:16.281894 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:16.281854    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq"]
Oct 30 01:56:16.282571 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:16.282538    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq" podUID="22765564-41a8-4a68-9bd6-9aff4d7a0655" containerName="maas-api" containerID="cri-o://36f400150e34203146a7c4760cad0f3ce4d469ab424700bb3b2ebeab39504520" gracePeriod=30
Oct 30 01:56:16.524700 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:16.524675    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq"
Oct 30 01:56:16.574255 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:16.574121    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22765564-41a8-4a68-9bd6-9aff4d7a0655-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22765564-41a8-4a68-9bd6-9aff4d7a0655-openshift-service-ca\") pod \"22765564-41a8-4a68-9bd6-9aff4d7a0655\" (UID: \"22765564-41a8-4a68-9bd6-9aff4d7a0655\") "
Oct 30 01:56:16.574255 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:16.574249    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22765564-41a8-4a68-9bd6-9aff4d7a0655-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22765564-41a8-4a68-9bd6-9aff4d7a0655-maas-api-tls\") pod \"22765564-41a8-4a68-9bd6-9aff4d7a0655\" (UID: \"22765564-41a8-4a68-9bd6-9aff4d7a0655\") "
Oct 30 01:56:16.574646 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:16.574290    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22765564-41a8-4a68-9bd6-9aff4d7a0655-metrics-certs\" (UniqueName: \"kubernetes.io/secret/22765564-41a8-4a68-9bd6-9aff4d7a0655-metrics-certs\") pod \"22765564-41a8-4a68-9bd6-9aff4d7a0655\" (UID: \"22765564-41a8-4a68-9bd6-9aff4d7a0655\") "
Oct 30 01:56:16.574646 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:16.574316    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22765564-41a8-4a68-9bd6-9aff4d7a0655-kube-api-access-8z8xr\" (UniqueName: \"kubernetes.io/projected/22765564-41a8-4a68-9bd6-9aff4d7a0655-kube-api-access-8z8xr\") pod \"22765564-41a8-4a68-9bd6-9aff4d7a0655\" (UID: \"22765564-41a8-4a68-9bd6-9aff4d7a0655\") "
Oct 30 01:56:16.574933 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:16.574879    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22765564-41a8-4a68-9bd6-9aff4d7a0655-openshift-service-ca" pod "22765564-41a8-4a68-9bd6-9aff4d7a0655" (UID: "22765564-41a8-4a68-9bd6-9aff4d7a0655"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:56:16.576658 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:16.576616    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22765564-41a8-4a68-9bd6-9aff4d7a0655-kube-api-access-8z8xr" pod "22765564-41a8-4a68-9bd6-9aff4d7a0655" (UID: "22765564-41a8-4a68-9bd6-9aff4d7a0655"). InnerVolumeSpecName "kube-api-access-8z8xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:56:16.576970 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:16.576941    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22765564-41a8-4a68-9bd6-9aff4d7a0655-metrics-certs" pod "22765564-41a8-4a68-9bd6-9aff4d7a0655" (UID: "22765564-41a8-4a68-9bd6-9aff4d7a0655"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:56:16.577113 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:16.576951    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22765564-41a8-4a68-9bd6-9aff4d7a0655-maas-api-tls" pod "22765564-41a8-4a68-9bd6-9aff4d7a0655" (UID: "22765564-41a8-4a68-9bd6-9aff4d7a0655"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:56:16.675476 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:16.675441    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22765564-41a8-4a68-9bd6-9aff4d7a0655-maas-api-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:56:16.675476 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:16.675470    4864 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22765564-41a8-4a68-9bd6-9aff4d7a0655-metrics-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:56:16.675476 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:16.675481    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z8xr\" (UniqueName: \"kubernetes.io/projected/22765564-41a8-4a68-9bd6-9aff4d7a0655-kube-api-access-8z8xr\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:56:16.675710 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:16.675491    4864 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22765564-41a8-4a68-9bd6-9aff4d7a0655-openshift-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:56:16.690363 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:16.690330    4864 generic.go:356] "Generic (PLEG): container finished" podID="22765564-41a8-4a68-9bd6-9aff4d7a0655" containerID="36f400150e34203146a7c4760cad0f3ce4d469ab424700bb3b2ebeab39504520" exitCode=0
Oct 30 01:56:16.690474 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:16.690374    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq"
Oct 30 01:56:16.690474 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:16.690418    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq" event={"ID":"22765564-41a8-4a68-9bd6-9aff4d7a0655","Type":"ContainerDied","Data":"36f400150e34203146a7c4760cad0f3ce4d469ab424700bb3b2ebeab39504520"}
Oct 30 01:56:16.690474 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:16.690471    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq" event={"ID":"22765564-41a8-4a68-9bd6-9aff4d7a0655","Type":"ContainerDied","Data":"44db66e150e52ae53e455a715ece3ae5c6e054e16f6bcbff79b1a1a2ff1bc07f"}
Oct 30 01:56:16.690601 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:16.690495    4864 scope.go:122] "RemoveContainer" containerID="36f400150e34203146a7c4760cad0f3ce4d469ab424700bb3b2ebeab39504520"
Oct 30 01:56:16.700812 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:16.700793    4864 scope.go:122] "RemoveContainer" containerID="36f400150e34203146a7c4760cad0f3ce4d469ab424700bb3b2ebeab39504520"
Oct 30 01:56:16.701143 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:56:16.701122    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36f400150e34203146a7c4760cad0f3ce4d469ab424700bb3b2ebeab39504520\": container with ID starting with 36f400150e34203146a7c4760cad0f3ce4d469ab424700bb3b2ebeab39504520 not found: ID does not exist" containerID="36f400150e34203146a7c4760cad0f3ce4d469ab424700bb3b2ebeab39504520"
Oct 30 01:56:16.701201 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:16.701152    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f400150e34203146a7c4760cad0f3ce4d469ab424700bb3b2ebeab39504520"} err="failed to get container status \"36f400150e34203146a7c4760cad0f3ce4d469ab424700bb3b2ebeab39504520\": rpc error: code = NotFound desc = could not find container \"36f400150e34203146a7c4760cad0f3ce4d469ab424700bb3b2ebeab39504520\": container with ID starting with 36f400150e34203146a7c4760cad0f3ce4d469ab424700bb3b2ebeab39504520 not found: ID does not exist"
Oct 30 01:56:16.736441 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:16.736414    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq"]
Oct 30 01:56:16.745505 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:16.745478    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1afb3e-5579675b65-lp6cq"]
Oct 30 01:56:17.792048 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:17.792016    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22765564-41a8-4a68-9bd6-9aff4d7a0655" path="/var/lib/kubelet/pods/22765564-41a8-4a68-9bd6-9aff4d7a0655/volumes"
Oct 30 01:56:19.038522 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:56:19.038483    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:56:28.693286 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:28.693259    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:56:28.700407 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:28.700380    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:56:49.041423 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:56:49.041383    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:56:59.691472 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:59.691427    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s"]
Oct 30 01:56:59.724356 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:56:59.724322    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx"]
Oct 30 01:57:04.692348 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:04.692291    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s" podUID="ef300502-05e8-43c9-99e3-657417491be0" containerName="payload-processing" containerID="cri-o://e9157a2510958064632209186ac8bf11b21bc9ef474a3fca62c6fc3d57cd0ad0" gracePeriod=25
Oct 30 01:57:04.725642 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:04.725604    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx" podUID="90c062b6-b706-4e09-92a2-5627352f1485" containerName="payload-pre-processing" containerID="cri-o://84e6f6ebb1a018a207e251ccbd71e921296bc52d655fbee1d2424625c62899a1" gracePeriod=25
Oct 30 01:57:09.883601 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:09.883563    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"]
Oct 30 01:57:09.883997 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:09.883788    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4" podUID="4d2b2d7c-c516-4da4-86c1-e958d87e0fb1" containerName="istio-proxy" containerID="cri-o://92195df5300aecb23e72f062aeccd9cf3cef27151b083583adc960a6098c560e" gracePeriod=30
Oct 30 01:57:10.898902 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:10.898866    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx" podUID="90c062b6-b706-4e09-92a2-5627352f1485" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:57:10.899334 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:10.898874    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s" podUID="ef300502-05e8-43c9-99e3-657417491be0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:57:15.142554 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.142527    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:57:15.241385 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.241349    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-workload-socket\") pod \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") "
Oct 30 01:57:15.241565 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.241430    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-podinfo\") pod \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") "
Oct 30 01:57:15.241565 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.241471    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-envoy\") pod \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") "
Oct 30 01:57:15.241565 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.241516    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-token\" (UniqueName: \"kubernetes.io/projected/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-token\") pod \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") "
Oct 30 01:57:15.241565 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.241555    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istiod-ca-cert\") pod \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") "
Oct 30 01:57:15.241794 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.241596    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-credential-socket\") pod \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") "
Oct 30 01:57:15.241794 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.241635    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-workload-certs\") pod \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") "
Oct 30 01:57:15.241794 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.241673    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-kube-api-access-5grfm\" (UniqueName: \"kubernetes.io/projected/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-kube-api-access-5grfm\") pod \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") "
Oct 30 01:57:15.241794 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.241679    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-workload-socket" pod "4d2b2d7c-c516-4da4-86c1-e958d87e0fb1" (UID: "4d2b2d7c-c516-4da4-86c1-e958d87e0fb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:57:15.242115 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.241773    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-data\") pod \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\" (UID: \"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1\") "
Oct 30 01:57:15.242115 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.241944    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-workload-certs" pod "4d2b2d7c-c516-4da4-86c1-e958d87e0fb1" (UID: "4d2b2d7c-c516-4da4-86c1-e958d87e0fb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:57:15.242115 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.241962    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-credential-socket" pod "4d2b2d7c-c516-4da4-86c1-e958d87e0fb1" (UID: "4d2b2d7c-c516-4da4-86c1-e958d87e0fb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:57:15.242115 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.241987    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istiod-ca-cert" pod "4d2b2d7c-c516-4da4-86c1-e958d87e0fb1" (UID: "4d2b2d7c-c516-4da4-86c1-e958d87e0fb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:57:15.242361 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.242185    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-data" pod "4d2b2d7c-c516-4da4-86c1-e958d87e0fb1" (UID: "4d2b2d7c-c516-4da4-86c1-e958d87e0fb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:57:15.242361 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.242333    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:15.242361 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.242358    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:15.242534 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.242371    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:15.242534 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.242387    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:15.242534 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.242399    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:15.243801 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.243771    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-podinfo" pod "4d2b2d7c-c516-4da4-86c1-e958d87e0fb1" (UID: "4d2b2d7c-c516-4da4-86c1-e958d87e0fb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:57:15.243934 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.243856    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-envoy" pod "4d2b2d7c-c516-4da4-86c1-e958d87e0fb1" (UID: "4d2b2d7c-c516-4da4-86c1-e958d87e0fb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:57:15.243934 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.243902    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-kube-api-access-5grfm" pod "4d2b2d7c-c516-4da4-86c1-e958d87e0fb1" (UID: "4d2b2d7c-c516-4da4-86c1-e958d87e0fb1"). InnerVolumeSpecName "kube-api-access-5grfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:57:15.244019 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.243967    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-token" pod "4d2b2d7c-c516-4da4-86c1-e958d87e0fb1" (UID: "4d2b2d7c-c516-4da4-86c1-e958d87e0fb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:57:15.323597 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.323561    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"]
Oct 30 01:57:15.341619 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.341571    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"]
Oct 30 01:57:15.343741 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.343708    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:15.343872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.343743    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:15.343872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.343758    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:15.343872 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.343772    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5grfm\" (UniqueName: \"kubernetes.io/projected/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1-kube-api-access-5grfm\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:15.992446 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.992403    4864 generic.go:356] "Generic (PLEG): container finished" podID="4d2b2d7c-c516-4da4-86c1-e958d87e0fb1" containerID="92195df5300aecb23e72f062aeccd9cf3cef27151b083583adc960a6098c560e" exitCode=0
Oct 30 01:57:15.992446 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.992431    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4" event={"ID":"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1","Type":"ContainerDied","Data":"92195df5300aecb23e72f062aeccd9cf3cef27151b083583adc960a6098c560e"}
Oct 30 01:57:15.992727 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.992458    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"
Oct 30 01:57:15.992727 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.992505    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4" event={"ID":"4d2b2d7c-c516-4da4-86c1-e958d87e0fb1","Type":"ContainerDied","Data":"2afa4cb05da8387d152baa8fb252608baec89a624243ad4ed738ae7ba147a4c7"}
Oct 30 01:57:15.992727 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:15.992536    4864 scope.go:122] "RemoveContainer" containerID="92195df5300aecb23e72f062aeccd9cf3cef27151b083583adc960a6098c560e"
Oct 30 01:57:16.001943 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:16.001923    4864 scope.go:122] "RemoveContainer" containerID="92195df5300aecb23e72f062aeccd9cf3cef27151b083583adc960a6098c560e"
Oct 30 01:57:16.002229 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:57:16.002207    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92195df5300aecb23e72f062aeccd9cf3cef27151b083583adc960a6098c560e\": container with ID starting with 92195df5300aecb23e72f062aeccd9cf3cef27151b083583adc960a6098c560e not found: ID does not exist" containerID="92195df5300aecb23e72f062aeccd9cf3cef27151b083583adc960a6098c560e"
Oct 30 01:57:16.002281 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:16.002235    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92195df5300aecb23e72f062aeccd9cf3cef27151b083583adc960a6098c560e"} err="failed to get container status \"92195df5300aecb23e72f062aeccd9cf3cef27151b083583adc960a6098c560e\": rpc error: code = NotFound desc = could not find container \"92195df5300aecb23e72f062aeccd9cf3cef27151b083583adc960a6098c560e\": container with ID starting with 92195df5300aecb23e72f062aeccd9cf3cef27151b083583adc960a6098c560e not found: ID does not exist"
Oct 30 01:57:16.053729 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:16.053660    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"]
Oct 30 01:57:16.063954 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:16.063930    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-bc127f-openshift-default-94fb98bf9-cfsk4"]
Oct 30 01:57:17.792783 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:17.792747    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d2b2d7c-c516-4da4-86c1-e958d87e0fb1" path="/var/lib/kubelet/pods/4d2b2d7c-c516-4da4-86c1-e958d87e0fb1/volumes"
Oct 30 01:57:19.038280 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:57:19.038240    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:57:20.898425 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:20.898391    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s" podUID="ef300502-05e8-43c9-99e3-657417491be0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:57:20.898822 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:20.898425    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx" podUID="90c062b6-b706-4e09-92a2-5627352f1485" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:57:24.978961 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:24.978931    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s"
Oct 30 01:57:25.000563 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.000538    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx"
Oct 30 01:57:25.034933 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.034897    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90c062b6-b706-4e09-92a2-5627352f1485-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c062b6-b706-4e09-92a2-5627352f1485-plugins-config-volume\") pod \"90c062b6-b706-4e09-92a2-5627352f1485\" (UID: \"90c062b6-b706-4e09-92a2-5627352f1485\") "
Oct 30 01:57:25.035102 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.035009    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef300502-05e8-43c9-99e3-657417491be0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef300502-05e8-43c9-99e3-657417491be0-plugins-config-volume\") pod \"ef300502-05e8-43c9-99e3-657417491be0\" (UID: \"ef300502-05e8-43c9-99e3-657417491be0\") "
Oct 30 01:57:25.035102 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.035051    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90c062b6-b706-4e09-92a2-5627352f1485-kube-api-access-2cx76\" (UniqueName: \"kubernetes.io/projected/90c062b6-b706-4e09-92a2-5627352f1485-kube-api-access-2cx76\") pod \"90c062b6-b706-4e09-92a2-5627352f1485\" (UID: \"90c062b6-b706-4e09-92a2-5627352f1485\") "
Oct 30 01:57:25.035197 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.035109    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef300502-05e8-43c9-99e3-657417491be0-kube-api-access-hpp79\" (UniqueName: \"kubernetes.io/projected/ef300502-05e8-43c9-99e3-657417491be0-kube-api-access-hpp79\") pod \"ef300502-05e8-43c9-99e3-657417491be0\" (UID: \"ef300502-05e8-43c9-99e3-657417491be0\") "
Oct 30 01:57:25.036046 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.036016    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef300502-05e8-43c9-99e3-657417491be0-plugins-config-volume" pod "ef300502-05e8-43c9-99e3-657417491be0" (UID: "ef300502-05e8-43c9-99e3-657417491be0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:57:25.036174 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.036116    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c062b6-b706-4e09-92a2-5627352f1485-plugins-config-volume" pod "90c062b6-b706-4e09-92a2-5627352f1485" (UID: "90c062b6-b706-4e09-92a2-5627352f1485"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:57:25.037927 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.037898    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef300502-05e8-43c9-99e3-657417491be0-kube-api-access-hpp79" pod "ef300502-05e8-43c9-99e3-657417491be0" (UID: "ef300502-05e8-43c9-99e3-657417491be0"). InnerVolumeSpecName "kube-api-access-hpp79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:57:25.038050 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.038033    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c062b6-b706-4e09-92a2-5627352f1485-kube-api-access-2cx76" pod "90c062b6-b706-4e09-92a2-5627352f1485" (UID: "90c062b6-b706-4e09-92a2-5627352f1485"). InnerVolumeSpecName "kube-api-access-2cx76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:57:25.041168 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.041140    4864 generic.go:356] "Generic (PLEG): container finished" podID="90c062b6-b706-4e09-92a2-5627352f1485" containerID="84e6f6ebb1a018a207e251ccbd71e921296bc52d655fbee1d2424625c62899a1" exitCode=0
Oct 30 01:57:25.041249 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.041184    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx"
Oct 30 01:57:25.041249 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.041214    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx" event={"ID":"90c062b6-b706-4e09-92a2-5627352f1485","Type":"ContainerDied","Data":"84e6f6ebb1a018a207e251ccbd71e921296bc52d655fbee1d2424625c62899a1"}
Oct 30 01:57:25.041366 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.041264    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx" event={"ID":"90c062b6-b706-4e09-92a2-5627352f1485","Type":"ContainerDied","Data":"fbb613556b33e0ae42e8757a647952792a8386b8642889e68b08862d7d3ce31e"}
Oct 30 01:57:25.041366 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.041285    4864 scope.go:122] "RemoveContainer" containerID="84e6f6ebb1a018a207e251ccbd71e921296bc52d655fbee1d2424625c62899a1"
Oct 30 01:57:25.042737 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.042713    4864 generic.go:356] "Generic (PLEG): container finished" podID="ef300502-05e8-43c9-99e3-657417491be0" containerID="e9157a2510958064632209186ac8bf11b21bc9ef474a3fca62c6fc3d57cd0ad0" exitCode=0
Oct 30 01:57:25.042848 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.042817    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s"
Oct 30 01:57:25.042915 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.042823    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s" event={"ID":"ef300502-05e8-43c9-99e3-657417491be0","Type":"ContainerDied","Data":"e9157a2510958064632209186ac8bf11b21bc9ef474a3fca62c6fc3d57cd0ad0"}
Oct 30 01:57:25.042979 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.042945    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s" event={"ID":"ef300502-05e8-43c9-99e3-657417491be0","Type":"ContainerDied","Data":"8d2e4ae5bb8492c36604a92ed52a6447b5860b7a34a2e9cda2eac80e0732352f"}
Oct 30 01:57:25.051147 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.051041    4864 scope.go:122] "RemoveContainer" containerID="84e6f6ebb1a018a207e251ccbd71e921296bc52d655fbee1d2424625c62899a1"
Oct 30 01:57:25.051606 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:57:25.051583    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e6f6ebb1a018a207e251ccbd71e921296bc52d655fbee1d2424625c62899a1\": container with ID starting with 84e6f6ebb1a018a207e251ccbd71e921296bc52d655fbee1d2424625c62899a1 not found: ID does not exist" containerID="84e6f6ebb1a018a207e251ccbd71e921296bc52d655fbee1d2424625c62899a1"
Oct 30 01:57:25.051719 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.051624    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e6f6ebb1a018a207e251ccbd71e921296bc52d655fbee1d2424625c62899a1"} err="failed to get container status \"84e6f6ebb1a018a207e251ccbd71e921296bc52d655fbee1d2424625c62899a1\": rpc error: code = NotFound desc = could not find container \"84e6f6ebb1a018a207e251ccbd71e921296bc52d655fbee1d2424625c62899a1\": container with ID starting with 84e6f6ebb1a018a207e251ccbd71e921296bc52d655fbee1d2424625c62899a1 not found: ID does not exist"
Oct 30 01:57:25.051719 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.051650    4864 scope.go:122] "RemoveContainer" containerID="e9157a2510958064632209186ac8bf11b21bc9ef474a3fca62c6fc3d57cd0ad0"
Oct 30 01:57:25.062712 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.062688    4864 scope.go:122] "RemoveContainer" containerID="e9157a2510958064632209186ac8bf11b21bc9ef474a3fca62c6fc3d57cd0ad0"
Oct 30 01:57:25.063026 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:57:25.063002    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9157a2510958064632209186ac8bf11b21bc9ef474a3fca62c6fc3d57cd0ad0\": container with ID starting with e9157a2510958064632209186ac8bf11b21bc9ef474a3fca62c6fc3d57cd0ad0 not found: ID does not exist" containerID="e9157a2510958064632209186ac8bf11b21bc9ef474a3fca62c6fc3d57cd0ad0"
Oct 30 01:57:25.063177 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.063031    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9157a2510958064632209186ac8bf11b21bc9ef474a3fca62c6fc3d57cd0ad0"} err="failed to get container status \"e9157a2510958064632209186ac8bf11b21bc9ef474a3fca62c6fc3d57cd0ad0\": rpc error: code = NotFound desc = could not find container \"e9157a2510958064632209186ac8bf11b21bc9ef474a3fca62c6fc3d57cd0ad0\": container with ID starting with e9157a2510958064632209186ac8bf11b21bc9ef474a3fca62c6fc3d57cd0ad0 not found: ID does not exist"
Oct 30 01:57:25.088875 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.088838    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s"]
Oct 30 01:57:25.091237 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.091213    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-bc127f-5c9c9c9684-wgb6s"]
Oct 30 01:57:25.121854 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.121819    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx"]
Oct 30 01:57:25.125762 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.125736    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bc127f-96f55c46c-6jggx"]
Oct 30 01:57:25.136474 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.136445    4864 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c062b6-b706-4e09-92a2-5627352f1485-plugins-config-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:25.136474 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.136471    4864 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef300502-05e8-43c9-99e3-657417491be0-plugins-config-volume\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:25.136602 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.136487    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cx76\" (UniqueName: \"kubernetes.io/projected/90c062b6-b706-4e09-92a2-5627352f1485-kube-api-access-2cx76\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:25.136602 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.136501    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpp79\" (UniqueName: \"kubernetes.io/projected/ef300502-05e8-43c9-99e3-657417491be0-kube-api-access-hpp79\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:25.792022 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.791988    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90c062b6-b706-4e09-92a2-5627352f1485" path="/var/lib/kubelet/pods/90c062b6-b706-4e09-92a2-5627352f1485/volumes"
Oct 30 01:57:25.792484 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:25.792463    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef300502-05e8-43c9-99e3-657417491be0" path="/var/lib/kubelet/pods/ef300502-05e8-43c9-99e3-657417491be0/volumes"
Oct 30 01:57:30.344165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.344121    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll" podUID="2388a847-21ff-46dd-8b45-4183a2448cfe" containerName="main" containerID="cri-o://5a8d3d1fed736a70ceabb889c1b3bf6c6e1af01cbc8dba83e609cfff8d6f9234" gracePeriod=45
Oct 30 01:57:30.361028 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.360971    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv" podUID="3c545315-888c-4418-8348-b06a0080a599" containerName="main" containerID="cri-o://b0d47757aff813bc96a7112adad5d899bb1544eccde69dad7d75065578426421" gracePeriod=45
Oct 30 01:57:30.616177 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.616150    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 01:57:30.642838 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.642812    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 01:57:30.686429 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.686394    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-home\" (UniqueName: \"kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-home\") pod \"3c545315-888c-4418-8348-b06a0080a599\" (UID: \"3c545315-888c-4418-8348-b06a0080a599\") "
Oct 30 01:57:30.686631 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.686444    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-tmp-dir\") pod \"2388a847-21ff-46dd-8b45-4183a2448cfe\" (UID: \"2388a847-21ff-46dd-8b45-4183a2448cfe\") "
Oct 30 01:57:30.686631 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.686493    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-dshm\") pod \"3c545315-888c-4418-8348-b06a0080a599\" (UID: \"3c545315-888c-4418-8348-b06a0080a599\") "
Oct 30 01:57:30.686631 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.686545    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-model-cache\") pod \"3c545315-888c-4418-8348-b06a0080a599\" (UID: \"3c545315-888c-4418-8348-b06a0080a599\") "
Oct 30 01:57:30.686631 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.686573    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-model-cache\") pod \"2388a847-21ff-46dd-8b45-4183a2448cfe\" (UID: \"2388a847-21ff-46dd-8b45-4183a2448cfe\") "
Oct 30 01:57:30.686631 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.686602    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2388a847-21ff-46dd-8b45-4183a2448cfe-tls-certs\" (UniqueName: \"kubernetes.io/secret/2388a847-21ff-46dd-8b45-4183a2448cfe-tls-certs\") pod \"2388a847-21ff-46dd-8b45-4183a2448cfe\" (UID: \"2388a847-21ff-46dd-8b45-4183a2448cfe\") "
Oct 30 01:57:30.686955 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.686660    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c545315-888c-4418-8348-b06a0080a599-kube-api-access-mqg7z\" (UniqueName: \"kubernetes.io/projected/3c545315-888c-4418-8348-b06a0080a599-kube-api-access-mqg7z\") pod \"3c545315-888c-4418-8348-b06a0080a599\" (UID: \"3c545315-888c-4418-8348-b06a0080a599\") "
Oct 30 01:57:30.686955 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.686725    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-tmp-dir\") pod \"3c545315-888c-4418-8348-b06a0080a599\" (UID: \"3c545315-888c-4418-8348-b06a0080a599\") "
Oct 30 01:57:30.686955 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.686736    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-home" pod "3c545315-888c-4418-8348-b06a0080a599" (UID: "3c545315-888c-4418-8348-b06a0080a599"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:57:30.686955 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.686776    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2388a847-21ff-46dd-8b45-4183a2448cfe-kube-api-access-hvjmn\" (UniqueName: \"kubernetes.io/projected/2388a847-21ff-46dd-8b45-4183a2448cfe-kube-api-access-hvjmn\") pod \"2388a847-21ff-46dd-8b45-4183a2448cfe\" (UID: \"2388a847-21ff-46dd-8b45-4183a2448cfe\") "
Oct 30 01:57:30.686955 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.686930    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c545315-888c-4418-8348-b06a0080a599-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c545315-888c-4418-8348-b06a0080a599-tls-certs\") pod \"3c545315-888c-4418-8348-b06a0080a599\" (UID: \"3c545315-888c-4418-8348-b06a0080a599\") "
Oct 30 01:57:30.687268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.686975    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-dshm\") pod \"2388a847-21ff-46dd-8b45-4183a2448cfe\" (UID: \"2388a847-21ff-46dd-8b45-4183a2448cfe\") "
Oct 30 01:57:30.687268 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.687009    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-home\" (UniqueName: \"kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-home\") pod \"2388a847-21ff-46dd-8b45-4183a2448cfe\" (UID: \"2388a847-21ff-46dd-8b45-4183a2448cfe\") "
Oct 30 01:57:30.687460 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.687413    4864 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-home\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:30.688131 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.686774    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-tmp-dir" pod "2388a847-21ff-46dd-8b45-4183a2448cfe" (UID: "2388a847-21ff-46dd-8b45-4183a2448cfe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:57:30.688131 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.687166    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-model-cache" pod "2388a847-21ff-46dd-8b45-4183a2448cfe" (UID: "2388a847-21ff-46dd-8b45-4183a2448cfe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:57:30.688131 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.687170    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-model-cache" pod "3c545315-888c-4418-8348-b06a0080a599" (UID: "3c545315-888c-4418-8348-b06a0080a599"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:57:30.688131 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.687619    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-home" pod "2388a847-21ff-46dd-8b45-4183a2448cfe" (UID: "2388a847-21ff-46dd-8b45-4183a2448cfe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:57:30.688131 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.688094    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-tmp-dir" pod "3c545315-888c-4418-8348-b06a0080a599" (UID: "3c545315-888c-4418-8348-b06a0080a599"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:57:30.689585 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.689560    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2388a847-21ff-46dd-8b45-4183a2448cfe-kube-api-access-hvjmn" pod "2388a847-21ff-46dd-8b45-4183a2448cfe" (UID: "2388a847-21ff-46dd-8b45-4183a2448cfe"). InnerVolumeSpecName "kube-api-access-hvjmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:57:30.689719 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.689671    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2388a847-21ff-46dd-8b45-4183a2448cfe-tls-certs" pod "2388a847-21ff-46dd-8b45-4183a2448cfe" (UID: "2388a847-21ff-46dd-8b45-4183a2448cfe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:57:30.689788 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.689764    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-dshm" pod "2388a847-21ff-46dd-8b45-4183a2448cfe" (UID: "2388a847-21ff-46dd-8b45-4183a2448cfe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:57:30.689986 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.689963    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-dshm" pod "3c545315-888c-4418-8348-b06a0080a599" (UID: "3c545315-888c-4418-8348-b06a0080a599"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:57:30.690061 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.690035    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c545315-888c-4418-8348-b06a0080a599-tls-certs" pod "3c545315-888c-4418-8348-b06a0080a599" (UID: "3c545315-888c-4418-8348-b06a0080a599"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:57:30.690675 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.690654    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c545315-888c-4418-8348-b06a0080a599-kube-api-access-mqg7z" pod "3c545315-888c-4418-8348-b06a0080a599" (UID: "3c545315-888c-4418-8348-b06a0080a599"). InnerVolumeSpecName "kube-api-access-mqg7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:57:30.788159 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.788124    4864 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-tmp-dir\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:30.788159 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.788154    4864 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-dshm\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:30.788159 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.788167    4864 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-model-cache\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:30.788416 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.788178    4864 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-model-cache\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:30.788416 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.788189    4864 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2388a847-21ff-46dd-8b45-4183a2448cfe-tls-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:30.788416 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.788201    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqg7z\" (UniqueName: \"kubernetes.io/projected/3c545315-888c-4418-8348-b06a0080a599-kube-api-access-mqg7z\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:30.788416 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.788212    4864 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c545315-888c-4418-8348-b06a0080a599-tmp-dir\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:30.788416 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.788224    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvjmn\" (UniqueName: \"kubernetes.io/projected/2388a847-21ff-46dd-8b45-4183a2448cfe-kube-api-access-hvjmn\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:30.788416 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.788235    4864 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c545315-888c-4418-8348-b06a0080a599-tls-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:30.788416 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.788246    4864 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-dshm\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:30.788416 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:30.788256    4864 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2388a847-21ff-46dd-8b45-4183a2448cfe-home\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:31.075552 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:31.075513    4864 generic.go:356] "Generic (PLEG): container finished" podID="2388a847-21ff-46dd-8b45-4183a2448cfe" containerID="5a8d3d1fed736a70ceabb889c1b3bf6c6e1af01cbc8dba83e609cfff8d6f9234" exitCode=0
Oct 30 01:57:31.075853 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:31.075559    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"
Oct 30 01:57:31.075853 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:31.075590    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll" event={"ID":"2388a847-21ff-46dd-8b45-4183a2448cfe","Type":"ContainerDied","Data":"5a8d3d1fed736a70ceabb889c1b3bf6c6e1af01cbc8dba83e609cfff8d6f9234"}
Oct 30 01:57:31.075853 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:31.075642    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll" event={"ID":"2388a847-21ff-46dd-8b45-4183a2448cfe","Type":"ContainerDied","Data":"c168ad2e8f2534bb42969b9e7e57ce971214ec67f63d9a94748d4bebf6e5fecb"}
Oct 30 01:57:31.075853 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:31.075667    4864 scope.go:122] "RemoveContainer" containerID="5a8d3d1fed736a70ceabb889c1b3bf6c6e1af01cbc8dba83e609cfff8d6f9234"
Oct 30 01:57:31.077002 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:31.076976    4864 generic.go:356] "Generic (PLEG): container finished" podID="3c545315-888c-4418-8348-b06a0080a599" containerID="b0d47757aff813bc96a7112adad5d899bb1544eccde69dad7d75065578426421" exitCode=0
Oct 30 01:57:31.077133 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:31.077029    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"
Oct 30 01:57:31.077133 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:31.077043    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv" event={"ID":"3c545315-888c-4418-8348-b06a0080a599","Type":"ContainerDied","Data":"b0d47757aff813bc96a7112adad5d899bb1544eccde69dad7d75065578426421"}
Oct 30 01:57:31.077133 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:31.077101    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv" event={"ID":"3c545315-888c-4418-8348-b06a0080a599","Type":"ContainerDied","Data":"2cc8784f938fb6217df39ffe422656e46f988f2778e92f6d276005bac9b3e7ce"}
Oct 30 01:57:31.084744 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:31.084725    4864 scope.go:122] "RemoveContainer" containerID="5a8d3d1fed736a70ceabb889c1b3bf6c6e1af01cbc8dba83e609cfff8d6f9234"
Oct 30 01:57:31.085031 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:57:31.085004    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a8d3d1fed736a70ceabb889c1b3bf6c6e1af01cbc8dba83e609cfff8d6f9234\": container with ID starting with 5a8d3d1fed736a70ceabb889c1b3bf6c6e1af01cbc8dba83e609cfff8d6f9234 not found: ID does not exist" containerID="5a8d3d1fed736a70ceabb889c1b3bf6c6e1af01cbc8dba83e609cfff8d6f9234"
Oct 30 01:57:31.085142 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:31.085041    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a8d3d1fed736a70ceabb889c1b3bf6c6e1af01cbc8dba83e609cfff8d6f9234"} err="failed to get container status \"5a8d3d1fed736a70ceabb889c1b3bf6c6e1af01cbc8dba83e609cfff8d6f9234\": rpc error: code = NotFound desc = could not find container \"5a8d3d1fed736a70ceabb889c1b3bf6c6e1af01cbc8dba83e609cfff8d6f9234\": container with ID starting with 5a8d3d1fed736a70ceabb889c1b3bf6c6e1af01cbc8dba83e609cfff8d6f9234 not found: ID does not exist"
Oct 30 01:57:31.085142 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:31.085067    4864 scope.go:122] "RemoveContainer" containerID="b0d47757aff813bc96a7112adad5d899bb1544eccde69dad7d75065578426421"
Oct 30 01:57:31.093283 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:31.093260    4864 scope.go:122] "RemoveContainer" containerID="b0d47757aff813bc96a7112adad5d899bb1544eccde69dad7d75065578426421"
Oct 30 01:57:31.093528 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:57:31.093511    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0d47757aff813bc96a7112adad5d899bb1544eccde69dad7d75065578426421\": container with ID starting with b0d47757aff813bc96a7112adad5d899bb1544eccde69dad7d75065578426421 not found: ID does not exist" containerID="b0d47757aff813bc96a7112adad5d899bb1544eccde69dad7d75065578426421"
Oct 30 01:57:31.093580 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:31.093534    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d47757aff813bc96a7112adad5d899bb1544eccde69dad7d75065578426421"} err="failed to get container status \"b0d47757aff813bc96a7112adad5d899bb1544eccde69dad7d75065578426421\": rpc error: code = NotFound desc = could not find container \"b0d47757aff813bc96a7112adad5d899bb1544eccde69dad7d75065578426421\": container with ID starting with b0d47757aff813bc96a7112adad5d899bb1544eccde69dad7d75065578426421 not found: ID does not exist"
Oct 30 01:57:31.121718 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:31.121587    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"]
Oct 30 01:57:31.124699 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:31.124675    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d5b2b8/distinct-w4-d5b2b8-kserve-8c4c7d88-8xgll"]
Oct 30 01:57:31.154195 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:31.154172    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"]
Oct 30 01:57:31.158247 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:31.158228    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d5b2b8/simulator-w4-d5b2b8-kserve-7dbcbcfbf6-bp4nv"]
Oct 30 01:57:31.792383 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:31.792350    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2388a847-21ff-46dd-8b45-4183a2448cfe" path="/var/lib/kubelet/pods/2388a847-21ff-46dd-8b45-4183a2448cfe/volumes"
Oct 30 01:57:31.792764 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:31.792752    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c545315-888c-4418-8348-b06a0080a599" path="/var/lib/kubelet/pods/3c545315-888c-4418-8348-b06a0080a599/volumes"
Oct 30 01:57:39.511256 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:39.511222    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78"]
Oct 30 01:57:39.511715 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:39.511440    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78" podUID="3f110028-043c-470d-9724-290fcf719e99" containerName="maas-api" containerID="cri-o://7dfea5dfb0f6d65f86a71b103422019e549fac4ac1f64c7ac3502e427cf35a22" gracePeriod=30
Oct 30 01:57:39.763945 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:39.763885    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78"
Oct 30 01:57:39.869595 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:39.869562    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f110028-043c-470d-9724-290fcf719e99-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f110028-043c-470d-9724-290fcf719e99-maas-api-tls\") pod \"3f110028-043c-470d-9724-290fcf719e99\" (UID: \"3f110028-043c-470d-9724-290fcf719e99\") "
Oct 30 01:57:39.869799 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:39.869606    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f110028-043c-470d-9724-290fcf719e99-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f110028-043c-470d-9724-290fcf719e99-openshift-service-ca\") pod \"3f110028-043c-470d-9724-290fcf719e99\" (UID: \"3f110028-043c-470d-9724-290fcf719e99\") "
Oct 30 01:57:39.869799 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:39.869637    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f110028-043c-470d-9724-290fcf719e99-kube-api-access-qgtw5\" (UniqueName: \"kubernetes.io/projected/3f110028-043c-470d-9724-290fcf719e99-kube-api-access-qgtw5\") pod \"3f110028-043c-470d-9724-290fcf719e99\" (UID: \"3f110028-043c-470d-9724-290fcf719e99\") "
Oct 30 01:57:39.869799 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:39.869665    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f110028-043c-470d-9724-290fcf719e99-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f110028-043c-470d-9724-290fcf719e99-metrics-certs\") pod \"3f110028-043c-470d-9724-290fcf719e99\" (UID: \"3f110028-043c-470d-9724-290fcf719e99\") "
Oct 30 01:57:39.870028 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:39.870001    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f110028-043c-470d-9724-290fcf719e99-openshift-service-ca" pod "3f110028-043c-470d-9724-290fcf719e99" (UID: "3f110028-043c-470d-9724-290fcf719e99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:57:39.871882 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:39.871857    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f110028-043c-470d-9724-290fcf719e99-metrics-certs" pod "3f110028-043c-470d-9724-290fcf719e99" (UID: "3f110028-043c-470d-9724-290fcf719e99"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:57:39.871966 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:39.871880    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f110028-043c-470d-9724-290fcf719e99-kube-api-access-qgtw5" pod "3f110028-043c-470d-9724-290fcf719e99" (UID: "3f110028-043c-470d-9724-290fcf719e99"). InnerVolumeSpecName "kube-api-access-qgtw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:57:39.871966 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:39.871884    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f110028-043c-470d-9724-290fcf719e99-maas-api-tls" pod "3f110028-043c-470d-9724-290fcf719e99" (UID: "3f110028-043c-470d-9724-290fcf719e99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:57:39.970538 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:39.970496    4864 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f110028-043c-470d-9724-290fcf719e99-maas-api-tls\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:39.970538 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:39.970534    4864 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f110028-043c-470d-9724-290fcf719e99-openshift-service-ca\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:39.970786 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:39.970551    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgtw5\" (UniqueName: \"kubernetes.io/projected/3f110028-043c-470d-9724-290fcf719e99-kube-api-access-qgtw5\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:39.970786 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:39.970563    4864 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f110028-043c-470d-9724-290fcf719e99-metrics-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:40.124869 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:40.124769    4864 generic.go:356] "Generic (PLEG): container finished" podID="3f110028-043c-470d-9724-290fcf719e99" containerID="7dfea5dfb0f6d65f86a71b103422019e549fac4ac1f64c7ac3502e427cf35a22" exitCode=0
Oct 30 01:57:40.124869 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:40.124826    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78"
Oct 30 01:57:40.124869 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:40.124855    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78" event={"ID":"3f110028-043c-470d-9724-290fcf719e99","Type":"ContainerDied","Data":"7dfea5dfb0f6d65f86a71b103422019e549fac4ac1f64c7ac3502e427cf35a22"}
Oct 30 01:57:40.125223 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:40.124921    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78" event={"ID":"3f110028-043c-470d-9724-290fcf719e99","Type":"ContainerDied","Data":"8b9953247357c4d29774d2f31f382ce5eab657256cb2814f0e8cce43662a31af"}
Oct 30 01:57:40.125223 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:40.124953    4864 scope.go:122] "RemoveContainer" containerID="7dfea5dfb0f6d65f86a71b103422019e549fac4ac1f64c7ac3502e427cf35a22"
Oct 30 01:57:40.134687 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:40.134670    4864 scope.go:122] "RemoveContainer" containerID="7dfea5dfb0f6d65f86a71b103422019e549fac4ac1f64c7ac3502e427cf35a22"
Oct 30 01:57:40.134940 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:57:40.134920    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dfea5dfb0f6d65f86a71b103422019e549fac4ac1f64c7ac3502e427cf35a22\": container with ID starting with 7dfea5dfb0f6d65f86a71b103422019e549fac4ac1f64c7ac3502e427cf35a22 not found: ID does not exist" containerID="7dfea5dfb0f6d65f86a71b103422019e549fac4ac1f64c7ac3502e427cf35a22"
Oct 30 01:57:40.134986 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:40.134947    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dfea5dfb0f6d65f86a71b103422019e549fac4ac1f64c7ac3502e427cf35a22"} err="failed to get container status \"7dfea5dfb0f6d65f86a71b103422019e549fac4ac1f64c7ac3502e427cf35a22\": rpc error: code = NotFound desc = could not find container \"7dfea5dfb0f6d65f86a71b103422019e549fac4ac1f64c7ac3502e427cf35a22\": container with ID starting with 7dfea5dfb0f6d65f86a71b103422019e549fac4ac1f64c7ac3502e427cf35a22 not found: ID does not exist"
Oct 30 01:57:40.170172 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:40.170139    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78"]
Oct 30 01:57:40.180629 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:40.180608    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d5b2b8-64d6cfddd4-csm78"]
Oct 30 01:57:41.791949 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:41.791913    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f110028-043c-470d-9724-290fcf719e99" path="/var/lib/kubelet/pods/3f110028-043c-470d-9724-290fcf719e99/volumes"
Oct 30 01:57:47.289959 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:47.289926    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"]
Oct 30 01:57:47.290374 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:47.290221    4864 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g" podUID="e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6" containerName="istio-proxy" containerID="cri-o://be243ed6cca81820e78ead50eedf580c7d4d595657fe9e914705ddaf37306ba9" gracePeriod=30
Oct 30 01:57:49.039886 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:57:49.039847    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:57:50.202572 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:50.202536    4864 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:57:50.202978 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:50.202593    4864 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g" podUID="e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:57:50.581792 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:50.581719    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-599679688f-rb5db_1cfebe80-0050-43c0-bff5-eecbfeb7ffb5/manager/0.log"
Oct 30 01:57:50.986994 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:50.986961    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nbrcn_145c8ccc-6041-44e5-8df0-c8c2420dfff2/manager/0.log"
Oct 30 01:57:52.261677 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.261642    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-tlc9c_62dcd375-7d44-4774-b18f-fb601a98b643/proxy/0.log"
Oct 30 01:57:52.524692 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.524631    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 01:57:52.589533 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.589501    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-token\" (UniqueName: \"kubernetes.io/projected/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-token\") pod \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") "
Oct 30 01:57:52.589714 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.589556    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-envoy\") pod \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") "
Oct 30 01:57:52.589714 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.589576    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istiod-ca-cert\") pod \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") "
Oct 30 01:57:52.589714 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.589608    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-credential-socket\") pod \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") "
Oct 30 01:57:52.589714 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.589644    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-podinfo\") pod \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") "
Oct 30 01:57:52.589714 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.589681    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-workload-certs\") pod \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") "
Oct 30 01:57:52.590013 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.589735    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-workload-socket\") pod \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") "
Oct 30 01:57:52.590013 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.589821    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-data\") pod \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") "
Oct 30 01:57:52.590013 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.589859    4864 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-kube-api-access-j45z8\" (UniqueName: \"kubernetes.io/projected/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-kube-api-access-j45z8\") pod \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\" (UID: \"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6\") "
Oct 30 01:57:52.590418 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.590094    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-workload-socket" pod "e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6" (UID: "e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:57:52.590418 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.590142    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istiod-ca-cert" pod "e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6" (UID: "e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:57:52.590418 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.590178    4864 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-workload-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:52.590637 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.590528    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-credential-socket" pod "e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6" (UID: "e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:57:52.591377 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.591347    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-workload-certs" pod "e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6" (UID: "e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:57:52.591971 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.591945    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-data" pod "e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6" (UID: "e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:57:52.592326 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.592199    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-envoy" pod "e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6" (UID: "e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:57:52.592326 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.592297    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-token" pod "e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6" (UID: "e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:57:52.592617 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.592592    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-kube-api-access-j45z8" pod "e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6" (UID: "e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6"). InnerVolumeSpecName "kube-api-access-j45z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:57:52.593674 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.593651    4864 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-podinfo" pod "e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6" (UID: "e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:57:52.691245 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.691212    4864 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-data\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:52.691245 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.691246    4864 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j45z8\" (UniqueName: \"kubernetes.io/projected/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-kube-api-access-j45z8\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:52.691452 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.691257    4864 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-token\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:52.691452 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.691278    4864 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-envoy\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:52.691452 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.691288    4864 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istiod-ca-cert\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:52.691452 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.691296    4864 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-credential-socket\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:52.691452 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.691307    4864 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-istio-podinfo\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:52.691452 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:52.691318    4864 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6-workload-certs\") on node \"ip-10-0-134-210.ec2.internal\" DevicePath \"\""
Oct 30 01:57:53.185316 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:53.185282    4864 generic.go:356] "Generic (PLEG): container finished" podID="e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6" containerID="be243ed6cca81820e78ead50eedf580c7d4d595657fe9e914705ddaf37306ba9" exitCode=0
Oct 30 01:57:53.185500 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:53.185327    4864 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"
Oct 30 01:57:53.185500 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:53.185370    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g" event={"ID":"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6","Type":"ContainerDied","Data":"be243ed6cca81820e78ead50eedf580c7d4d595657fe9e914705ddaf37306ba9"}
Oct 30 01:57:53.185500 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:53.185428    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g" event={"ID":"e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6","Type":"ContainerDied","Data":"a5c3c1d7cdeadcbdc5c47baaa6ac997ccfdf205f2b96dea338cb6be9e15ff0c4"}
Oct 30 01:57:53.185500 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:53.185451    4864 scope.go:122] "RemoveContainer" containerID="be243ed6cca81820e78ead50eedf580c7d4d595657fe9e914705ddaf37306ba9"
Oct 30 01:57:53.195138 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:53.195117    4864 scope.go:122] "RemoveContainer" containerID="be243ed6cca81820e78ead50eedf580c7d4d595657fe9e914705ddaf37306ba9"
Oct 30 01:57:53.195402 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:57:53.195385    4864 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be243ed6cca81820e78ead50eedf580c7d4d595657fe9e914705ddaf37306ba9\": container with ID starting with be243ed6cca81820e78ead50eedf580c7d4d595657fe9e914705ddaf37306ba9 not found: ID does not exist" containerID="be243ed6cca81820e78ead50eedf580c7d4d595657fe9e914705ddaf37306ba9"
Oct 30 01:57:53.195460 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:53.195412    4864 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be243ed6cca81820e78ead50eedf580c7d4d595657fe9e914705ddaf37306ba9"} err="failed to get container status \"be243ed6cca81820e78ead50eedf580c7d4d595657fe9e914705ddaf37306ba9\": rpc error: code = NotFound desc = could not find container \"be243ed6cca81820e78ead50eedf580c7d4d595657fe9e914705ddaf37306ba9\": container with ID starting with be243ed6cca81820e78ead50eedf580c7d4d595657fe9e914705ddaf37306ba9 not found: ID does not exist"
Oct 30 01:57:53.233988 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:53.233958    4864 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"]
Oct 30 01:57:53.240185 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:53.240162    4864 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-wwb7g"]
Oct 30 01:57:53.791495 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:57:53.791461    4864 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6" path="/var/lib/kubelet/pods/e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6/volumes"
Oct 30 01:58:03.076624 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:03.076591    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-599679688f-rb5db_1cfebe80-0050-43c0-bff5-eecbfeb7ffb5/manager/0.log"
Oct 30 01:58:03.475366 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:03.475332    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nbrcn_145c8ccc-6041-44e5-8df0-c8c2420dfff2/manager/0.log"
Oct 30 01:58:04.800758 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:04.800720    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-tlc9c_62dcd375-7d44-4774-b18f-fb601a98b643/proxy/0.log"
Oct 30 01:58:05.515881 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:05.515849    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6d5c8bc49-t2rj5_8bc51d8f-34a6-4134-9033-90fb1c5cabc5/maas-api/0.log"
Oct 30 01:58:11.728416 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:11.728382    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57cd69bbcd-76gbt_6bd30c09-5438-468b-b353-eb0a1ba483b5/kube-auth-proxy/0.log"
Oct 30 01:58:13.161116 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:13.161060    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-w45z8_a13e1eb5-b23d-49ac-9e1d-6b431e9f07fd/payload-pre-processing/0.log"
Oct 30 01:58:13.343009 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:13.342981    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6k58k_d2a355c3-0703-422f-8ddb-555c4a364ade/payload-processing/0.log"
Oct 30 01:58:13.536463 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:13.536432    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-94c9fcc47-fdtjt_d95535e9-e677-463a-a662-7e8045c71d9b/router/0.log"
Oct 30 01:58:14.267011 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:14.266977    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-c22wx_bdd5e277-4012-4026-896c-89c2aa9db931/main/0.log"
Oct 30 01:58:14.449144 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:14.449114    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-q58ws_7ce6cfb5-72b1-48d2-8311-10c69cf5dd2c/main/0.log"
Oct 30 01:58:14.654424 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:14.654327    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-6c89k_ec811c9f-eaa8-4c9b-8b96-8b690fb63772/main/0.log"
Oct 30 01:58:15.385699 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:15.385664    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-vc5ss_a3c75ce0-9007-4481-a55a-e04106a30699/main/0.log"
Oct 30 01:58:15.562216 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:15.562182    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-l9blh_6ba3a068-e4af-4bd1-b007-2f73f455f65c/main/0.log"
Oct 30 01:58:19.040427 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:58:19.040382    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:58:25.528519 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.528484    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qqxng/must-gather-zzf7x"]
Oct 30 01:58:25.528961 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.528911    4864 cpu_manager.go:414] "containerMap: removing container" podUID="3c545315-888c-4418-8348-b06a0080a599" containerName="main"
Oct 30 01:58:25.528961 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.528931    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c545315-888c-4418-8348-b06a0080a599" containerName="main"
Oct 30 01:58:25.528961 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.528950    4864 cpu_manager.go:414] "containerMap: removing container" podUID="3f110028-043c-470d-9724-290fcf719e99" containerName="maas-api"
Oct 30 01:58:25.528961 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.528958    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f110028-043c-470d-9724-290fcf719e99" containerName="maas-api"
Oct 30 01:58:25.529165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.528971    4864 cpu_manager.go:414] "containerMap: removing container" podUID="19b52d59-2121-4a46-b9a4-8c80742be6c3" containerName="maas-api"
Oct 30 01:58:25.529165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.528979    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b52d59-2121-4a46-b9a4-8c80742be6c3" containerName="maas-api"
Oct 30 01:58:25.529165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.528996    4864 cpu_manager.go:414] "containerMap: removing container" podUID="46b674ae-7af3-4ef9-a04a-bea2b1071c1d" containerName="payload-pre-processing"
Oct 30 01:58:25.529165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529003    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46b674ae-7af3-4ef9-a04a-bea2b1071c1d" containerName="payload-pre-processing"
Oct 30 01:58:25.529165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529014    4864 cpu_manager.go:414] "containerMap: removing container" podUID="41fd0656-df2b-4cd7-9f6a-ac93a18f9579" containerName="istio-proxy"
Oct 30 01:58:25.529165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529021    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41fd0656-df2b-4cd7-9f6a-ac93a18f9579" containerName="istio-proxy"
Oct 30 01:58:25.529165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529032    4864 cpu_manager.go:414] "containerMap: removing container" podUID="eab47e5b-e868-483e-92a2-e3414cc1dda2" containerName="istio-proxy"
Oct 30 01:58:25.529165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529039    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eab47e5b-e868-483e-92a2-e3414cc1dda2" containerName="istio-proxy"
Oct 30 01:58:25.529165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529060    4864 cpu_manager.go:414] "containerMap: removing container" podUID="90c062b6-b706-4e09-92a2-5627352f1485" containerName="payload-pre-processing"
Oct 30 01:58:25.529165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529067    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90c062b6-b706-4e09-92a2-5627352f1485" containerName="payload-pre-processing"
Oct 30 01:58:25.529165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529091    4864 cpu_manager.go:414] "containerMap: removing container" podUID="e8667b92-fdd3-49f8-bd1b-541d378530ef" containerName="main"
Oct 30 01:58:25.529165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529096    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8667b92-fdd3-49f8-bd1b-541d378530ef" containerName="main"
Oct 30 01:58:25.529165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529105    4864 cpu_manager.go:414] "containerMap: removing container" podUID="778281fb-fe1b-4f27-bd43-62920826a0f9" containerName="istio-proxy"
Oct 30 01:58:25.529165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529109    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="778281fb-fe1b-4f27-bd43-62920826a0f9" containerName="istio-proxy"
Oct 30 01:58:25.529165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529116    4864 cpu_manager.go:414] "containerMap: removing container" podUID="ef300502-05e8-43c9-99e3-657417491be0" containerName="payload-processing"
Oct 30 01:58:25.529165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529121    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef300502-05e8-43c9-99e3-657417491be0" containerName="payload-processing"
Oct 30 01:58:25.529165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529126    4864 cpu_manager.go:414] "containerMap: removing container" podUID="e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c" containerName="maas-api"
Oct 30 01:58:25.529165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529131    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c" containerName="maas-api"
Oct 30 01:58:25.529165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529138    4864 cpu_manager.go:414] "containerMap: removing container" podUID="83de80bb-1c42-4450-9019-abc0e614280b" containerName="payload-processing"
Oct 30 01:58:25.529165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529143    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83de80bb-1c42-4450-9019-abc0e614280b" containerName="payload-processing"
Oct 30 01:58:25.529165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529149    4864 cpu_manager.go:414] "containerMap: removing container" podUID="e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6" containerName="istio-proxy"
Oct 30 01:58:25.529165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529153    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6" containerName="istio-proxy"
Oct 30 01:58:25.529165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529165    4864 cpu_manager.go:414] "containerMap: removing container" podUID="4d2b2d7c-c516-4da4-86c1-e958d87e0fb1" containerName="istio-proxy"
Oct 30 01:58:25.529165 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529170    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d2b2d7c-c516-4da4-86c1-e958d87e0fb1" containerName="istio-proxy"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529179    4864 cpu_manager.go:414] "containerMap: removing container" podUID="2388a847-21ff-46dd-8b45-4183a2448cfe" containerName="main"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529183    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2388a847-21ff-46dd-8b45-4183a2448cfe" containerName="main"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529195    4864 cpu_manager.go:414] "containerMap: removing container" podUID="22765564-41a8-4a68-9bd6-9aff4d7a0655" containerName="maas-api"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529199    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22765564-41a8-4a68-9bd6-9aff4d7a0655" containerName="maas-api"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529207    4864 cpu_manager.go:414] "containerMap: removing container" podUID="53ecb5dd-2b7f-4139-8463-06583186f2e0" containerName="maas-api"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529211    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53ecb5dd-2b7f-4139-8463-06583186f2e0" containerName="maas-api"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529285    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="90c062b6-b706-4e09-92a2-5627352f1485" containerName="payload-pre-processing"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529295    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8667b92-fdd3-49f8-bd1b-541d378530ef" containerName="main"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529302    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef300502-05e8-43c9-99e3-657417491be0" containerName="payload-processing"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529307    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ceca67-73d9-4d66-8fbf-c5af36ec1e7c" containerName="maas-api"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529312    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="83de80bb-1c42-4450-9019-abc0e614280b" containerName="payload-processing"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529318    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64b0b95-75f6-4d2a-9a26-dc73d2c53ea6" containerName="istio-proxy"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529325    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d2b2d7c-c516-4da4-86c1-e958d87e0fb1" containerName="istio-proxy"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529330    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="2388a847-21ff-46dd-8b45-4183a2448cfe" containerName="main"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529336    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="41fd0656-df2b-4cd7-9f6a-ac93a18f9579" containerName="istio-proxy"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529342    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="22765564-41a8-4a68-9bd6-9aff4d7a0655" containerName="maas-api"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529348    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="53ecb5dd-2b7f-4139-8463-06583186f2e0" containerName="maas-api"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529353    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="778281fb-fe1b-4f27-bd43-62920826a0f9" containerName="istio-proxy"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529358    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c545315-888c-4418-8348-b06a0080a599" containerName="main"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529363    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f110028-043c-470d-9724-290fcf719e99" containerName="maas-api"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529368    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b52d59-2121-4a46-b9a4-8c80742be6c3" containerName="maas-api"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529373    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="46b674ae-7af3-4ef9-a04a-bea2b1071c1d" containerName="payload-pre-processing"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529378    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="41fd0656-df2b-4cd7-9f6a-ac93a18f9579" containerName="istio-proxy"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529383    4864 memory_manager.go:365] "RemoveStaleState removing state" podUID="eab47e5b-e868-483e-92a2-e3414cc1dda2" containerName="istio-proxy"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529444    4864 cpu_manager.go:414] "containerMap: removing container" podUID="41fd0656-df2b-4cd7-9f6a-ac93a18f9579" containerName="istio-proxy"
Oct 30 01:58:25.529901 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.529448    4864 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41fd0656-df2b-4cd7-9f6a-ac93a18f9579" containerName="istio-proxy"
Oct 30 01:58:25.532530 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.532513    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qqxng/must-gather-zzf7x"
Oct 30 01:58:25.545796 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.545775    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qqxng\"/\"kube-root-ca.crt\""
Oct 30 01:58:25.556301 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.556278    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qqxng\"/\"openshift-service-ca.crt\""
Oct 30 01:58:25.558976 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.558953    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qqxng\"/\"default-dockercfg-jkqcz\""
Oct 30 01:58:25.558976 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.558965    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qqxng/must-gather-zzf7x"]
Oct 30 01:58:25.593730 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.593694    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47k4p\" (UniqueName: \"kubernetes.io/projected/917ffd85-cd3e-42ae-ab07-279fbcb4ec21-kube-api-access-47k4p\") pod \"must-gather-zzf7x\" (UID: \"917ffd85-cd3e-42ae-ab07-279fbcb4ec21\") " pod="openshift-must-gather-qqxng/must-gather-zzf7x"
Oct 30 01:58:25.593868 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.593838    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/917ffd85-cd3e-42ae-ab07-279fbcb4ec21-must-gather-output\") pod \"must-gather-zzf7x\" (UID: \"917ffd85-cd3e-42ae-ab07-279fbcb4ec21\") " pod="openshift-must-gather-qqxng/must-gather-zzf7x"
Oct 30 01:58:25.632131 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.632098    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vgwmw/must-gather-wxzh7"]
Oct 30 01:58:25.636240 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.636222    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kqdbw/must-gather-sqq2l"]
Oct 30 01:58:25.636365 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.636349    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vgwmw/must-gather-wxzh7"
Oct 30 01:58:25.639517 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.639498    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kqdbw/must-gather-sqq2l"
Oct 30 01:58:25.656311 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.656291    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vgwmw/must-gather-wxzh7"]
Oct 30 01:58:25.660166 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.660143    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vgwmw\"/\"kube-root-ca.crt\""
Oct 30 01:58:25.661718 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.661695    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kqdbw/must-gather-sqq2l"]
Oct 30 01:58:25.664884 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.664863    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vgwmw\"/\"default-dockercfg-j6c5x\""
Oct 30 01:58:25.664995 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.664901    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vgwmw\"/\"openshift-service-ca.crt\""
Oct 30 01:58:25.664995 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.664940    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kqdbw\"/\"kube-root-ca.crt\""
Oct 30 01:58:25.664995 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.664941    4864 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kqdbw\"/\"openshift-service-ca.crt\""
Oct 30 01:58:25.694328 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.694298    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8txh\" (UniqueName: \"kubernetes.io/projected/1b09bfdd-3763-48ec-91eb-3799ed2e3dfb-kube-api-access-j8txh\") pod \"must-gather-wxzh7\" (UID: \"1b09bfdd-3763-48ec-91eb-3799ed2e3dfb\") " pod="openshift-must-gather-vgwmw/must-gather-wxzh7"
Oct 30 01:58:25.694455 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.694347    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/917ffd85-cd3e-42ae-ab07-279fbcb4ec21-must-gather-output\") pod \"must-gather-zzf7x\" (UID: \"917ffd85-cd3e-42ae-ab07-279fbcb4ec21\") " pod="openshift-must-gather-qqxng/must-gather-zzf7x"
Oct 30 01:58:25.694455 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.694375    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/30ff5a8b-c19c-478d-9904-c5abdf4f260f-must-gather-output\") pod \"must-gather-sqq2l\" (UID: \"30ff5a8b-c19c-478d-9904-c5abdf4f260f\") " pod="openshift-must-gather-kqdbw/must-gather-sqq2l"
Oct 30 01:58:25.694574 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.694522    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w7wx\" (UniqueName: \"kubernetes.io/projected/30ff5a8b-c19c-478d-9904-c5abdf4f260f-kube-api-access-7w7wx\") pod \"must-gather-sqq2l\" (UID: \"30ff5a8b-c19c-478d-9904-c5abdf4f260f\") " pod="openshift-must-gather-kqdbw/must-gather-sqq2l"
Oct 30 01:58:25.694574 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.694550    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1b09bfdd-3763-48ec-91eb-3799ed2e3dfb-must-gather-output\") pod \"must-gather-wxzh7\" (UID: \"1b09bfdd-3763-48ec-91eb-3799ed2e3dfb\") " pod="openshift-must-gather-vgwmw/must-gather-wxzh7"
Oct 30 01:58:25.694692 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.694576    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47k4p\" (UniqueName: \"kubernetes.io/projected/917ffd85-cd3e-42ae-ab07-279fbcb4ec21-kube-api-access-47k4p\") pod \"must-gather-zzf7x\" (UID: \"917ffd85-cd3e-42ae-ab07-279fbcb4ec21\") " pod="openshift-must-gather-qqxng/must-gather-zzf7x"
Oct 30 01:58:25.694786 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.694768    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/917ffd85-cd3e-42ae-ab07-279fbcb4ec21-must-gather-output\") pod \"must-gather-zzf7x\" (UID: \"917ffd85-cd3e-42ae-ab07-279fbcb4ec21\") " pod="openshift-must-gather-qqxng/must-gather-zzf7x"
Oct 30 01:58:25.719943 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.719916    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47k4p\" (UniqueName: \"kubernetes.io/projected/917ffd85-cd3e-42ae-ab07-279fbcb4ec21-kube-api-access-47k4p\") pod \"must-gather-zzf7x\" (UID: \"917ffd85-cd3e-42ae-ab07-279fbcb4ec21\") " pod="openshift-must-gather-qqxng/must-gather-zzf7x"
Oct 30 01:58:25.759708 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.759682    4864 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kqdbw\"/\"default-dockercfg-np4lm\""
Oct 30 01:58:25.795645 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.795574    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w7wx\" (UniqueName: \"kubernetes.io/projected/30ff5a8b-c19c-478d-9904-c5abdf4f260f-kube-api-access-7w7wx\") pod \"must-gather-sqq2l\" (UID: \"30ff5a8b-c19c-478d-9904-c5abdf4f260f\") " pod="openshift-must-gather-kqdbw/must-gather-sqq2l"
Oct 30 01:58:25.795645 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.795611    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1b09bfdd-3763-48ec-91eb-3799ed2e3dfb-must-gather-output\") pod \"must-gather-wxzh7\" (UID: \"1b09bfdd-3763-48ec-91eb-3799ed2e3dfb\") " pod="openshift-must-gather-vgwmw/must-gather-wxzh7"
Oct 30 01:58:25.795842 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.795691    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8txh\" (UniqueName: \"kubernetes.io/projected/1b09bfdd-3763-48ec-91eb-3799ed2e3dfb-kube-api-access-j8txh\") pod \"must-gather-wxzh7\" (UID: \"1b09bfdd-3763-48ec-91eb-3799ed2e3dfb\") " pod="openshift-must-gather-vgwmw/must-gather-wxzh7"
Oct 30 01:58:25.795842 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.795728    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/30ff5a8b-c19c-478d-9904-c5abdf4f260f-must-gather-output\") pod \"must-gather-sqq2l\" (UID: \"30ff5a8b-c19c-478d-9904-c5abdf4f260f\") " pod="openshift-must-gather-kqdbw/must-gather-sqq2l"
Oct 30 01:58:25.796026 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.796006    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1b09bfdd-3763-48ec-91eb-3799ed2e3dfb-must-gather-output\") pod \"must-gather-wxzh7\" (UID: \"1b09bfdd-3763-48ec-91eb-3799ed2e3dfb\") " pod="openshift-must-gather-vgwmw/must-gather-wxzh7"
Oct 30 01:58:25.796145 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.796102    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/30ff5a8b-c19c-478d-9904-c5abdf4f260f-must-gather-output\") pod \"must-gather-sqq2l\" (UID: \"30ff5a8b-c19c-478d-9904-c5abdf4f260f\") " pod="openshift-must-gather-kqdbw/must-gather-sqq2l"
Oct 30 01:58:25.820694 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.820663    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8txh\" (UniqueName: \"kubernetes.io/projected/1b09bfdd-3763-48ec-91eb-3799ed2e3dfb-kube-api-access-j8txh\") pod \"must-gather-wxzh7\" (UID: \"1b09bfdd-3763-48ec-91eb-3799ed2e3dfb\") " pod="openshift-must-gather-vgwmw/must-gather-wxzh7"
Oct 30 01:58:25.820787 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.820663    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w7wx\" (UniqueName: \"kubernetes.io/projected/30ff5a8b-c19c-478d-9904-c5abdf4f260f-kube-api-access-7w7wx\") pod \"must-gather-sqq2l\" (UID: \"30ff5a8b-c19c-478d-9904-c5abdf4f260f\") " pod="openshift-must-gather-kqdbw/must-gather-sqq2l"
Oct 30 01:58:25.846530 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.846507    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qqxng/must-gather-zzf7x"
Oct 30 01:58:25.951539 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.951506    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vgwmw/must-gather-wxzh7"
Oct 30 01:58:25.955799 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:25.955777    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kqdbw/must-gather-sqq2l"
Oct 30 01:58:26.145058 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:26.144959    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qqxng/must-gather-zzf7x"]
Oct 30 01:58:26.149277 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:58:26.149252    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod917ffd85_cd3e_42ae_ab07_279fbcb4ec21.slice/crio-7ef4cb4f51855f07a6a9239258b17b6d7c6d836d8b846e34cadedbc4b62251ff WatchSource:0}: Error finding container 7ef4cb4f51855f07a6a9239258b17b6d7c6d836d8b846e34cadedbc4b62251ff: Status 404 returned error can't find the container with id 7ef4cb4f51855f07a6a9239258b17b6d7c6d836d8b846e34cadedbc4b62251ff
Oct 30 01:58:26.151086 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:26.151052    4864 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:58:26.257365 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:26.257315    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kqdbw/must-gather-sqq2l"]
Oct 30 01:58:26.259888 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:58:26.259863    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30ff5a8b_c19c_478d_9904_c5abdf4f260f.slice/crio-8d122a8cc928020cdc326037b975df65dc646bcad7f197bbb3787d78686ecb72 WatchSource:0}: Error finding container 8d122a8cc928020cdc326037b975df65dc646bcad7f197bbb3787d78686ecb72: Status 404 returned error can't find the container with id 8d122a8cc928020cdc326037b975df65dc646bcad7f197bbb3787d78686ecb72
Oct 30 01:58:26.267700 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:26.267677    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vgwmw/must-gather-wxzh7"]
Oct 30 01:58:26.270397 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:58:26.270370    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b09bfdd_3763_48ec_91eb_3799ed2e3dfb.slice/crio-a1600569897407d777ac3e1cfd6d18e73af4c4b76177d478a13d8b5c09d1c3a9 WatchSource:0}: Error finding container a1600569897407d777ac3e1cfd6d18e73af4c4b76177d478a13d8b5c09d1c3a9: Status 404 returned error can't find the container with id a1600569897407d777ac3e1cfd6d18e73af4c4b76177d478a13d8b5c09d1c3a9
Oct 30 01:58:26.339706 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:26.339669    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qqxng/must-gather-zzf7x" event={"ID":"917ffd85-cd3e-42ae-ab07-279fbcb4ec21","Type":"ContainerStarted","Data":"7ef4cb4f51855f07a6a9239258b17b6d7c6d836d8b846e34cadedbc4b62251ff"}
Oct 30 01:58:26.340851 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:26.340824    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kqdbw/must-gather-sqq2l" event={"ID":"30ff5a8b-c19c-478d-9904-c5abdf4f260f","Type":"ContainerStarted","Data":"8d122a8cc928020cdc326037b975df65dc646bcad7f197bbb3787d78686ecb72"}
Oct 30 01:58:26.341921 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:26.341898    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vgwmw/must-gather-wxzh7" event={"ID":"1b09bfdd-3763-48ec-91eb-3799ed2e3dfb","Type":"ContainerStarted","Data":"a1600569897407d777ac3e1cfd6d18e73af4c4b76177d478a13d8b5c09d1c3a9"}
Oct 30 01:58:27.362905 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:27.362818    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qqxng/must-gather-zzf7x" event={"ID":"917ffd85-cd3e-42ae-ab07-279fbcb4ec21","Type":"ContainerStarted","Data":"141c4e70409a5d29b708d9da64a4aa3422e4277a54d37766082eb0fefa70adae"}
Oct 30 01:58:27.362905 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:27.362882    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qqxng/must-gather-zzf7x" event={"ID":"917ffd85-cd3e-42ae-ab07-279fbcb4ec21","Type":"ContainerStarted","Data":"364361da0e63a87c0dd2ef02b16a1667ca5134aa0bad66d518119a25224a288d"}
Oct 30 01:58:27.402296 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:27.402240    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-qqxng/must-gather-zzf7x" podStartSLOduration=1.758285004 podStartE2EDuration="2.402223509s" totalImagesPullingTime="643.938505ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:27.401268766 +0000 UTC m=+4619.739370678" watchObservedRunningTime="2026-10-30 01:58:27.402223509 +0000 UTC m=+4619.740325420"
Oct 30 01:58:31.213976 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:31.213930    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wk65n_bbef968a-7aa4-4e23-9278-3706e60d8c0b/global-pull-secret-syncer/0.log"
Oct 30 01:58:31.402611 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:31.402578    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6q8k5_4509811e-30a0-4c8b-b21e-58826bd1f6c6/connection-checker/0.log"
Oct 30 01:58:31.525439 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:31.525368    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-dtn28_a2dadf26-b255-4281-91f4-98536d370d21/connection-checker/0.log"
Oct 30 01:58:31.648791 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:31.648759    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-pq5xl_fc3514a7-8b72-40ff-b45a-1c2005b59360/connection-checker/0.log"
Oct 30 01:58:32.097758 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:32.097676    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-77dcb_2ac86565-2bfb-45e4-840d-c0f3a29cdfa5/konnectivity-agent/0.log"
Oct 30 01:58:32.383141 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:32.382349    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-210.ec2.internal_17edaf9736d7ee1e0409a6d573ccb434/haproxy/0.log"
Oct 30 01:58:42.549091 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:42.548998    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kqdbw/must-gather-sqq2l" event={"ID":"30ff5a8b-c19c-478d-9904-c5abdf4f260f","Type":"ContainerStarted","Data":"9f82ada5b6a71bd6d4ce3ffbde97e2be4ebe122e6bbeddf3ea47062a16a7edb3"}
Oct 30 01:58:42.553816 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:42.550525    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vgwmw/must-gather-wxzh7" event={"ID":"1b09bfdd-3763-48ec-91eb-3799ed2e3dfb","Type":"ContainerStarted","Data":"d4531ad907c253ec5f5fc162744e3b22f463a87a43739a5a83448743975e02dc"}
Oct 30 01:58:43.260942 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:43.260905    4864 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64"]
Oct 30 01:58:43.268741 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:43.266978    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64"
Oct 30 01:58:43.288401 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:43.288365    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64"]
Oct 30 01:58:43.354886 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:43.354419    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0f185f9b-7950-4eed-8f05-c410e3cb57ba-proc\") pod \"perf-node-gather-daemonset-k9w64\" (UID: \"0f185f9b-7950-4eed-8f05-c410e3cb57ba\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64"
Oct 30 01:58:43.354886 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:43.354551    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f185f9b-7950-4eed-8f05-c410e3cb57ba-lib-modules\") pod \"perf-node-gather-daemonset-k9w64\" (UID: \"0f185f9b-7950-4eed-8f05-c410e3cb57ba\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64"
Oct 30 01:58:43.354886 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:43.354630    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0f185f9b-7950-4eed-8f05-c410e3cb57ba-podres\") pod \"perf-node-gather-daemonset-k9w64\" (UID: \"0f185f9b-7950-4eed-8f05-c410e3cb57ba\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64"
Oct 30 01:58:43.354886 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:43.354679    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f185f9b-7950-4eed-8f05-c410e3cb57ba-sys\") pod \"perf-node-gather-daemonset-k9w64\" (UID: \"0f185f9b-7950-4eed-8f05-c410e3cb57ba\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64"
Oct 30 01:58:43.354886 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:43.354710    4864 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7tfd\" (UniqueName: \"kubernetes.io/projected/0f185f9b-7950-4eed-8f05-c410e3cb57ba-kube-api-access-q7tfd\") pod \"perf-node-gather-daemonset-k9w64\" (UID: \"0f185f9b-7950-4eed-8f05-c410e3cb57ba\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64"
Oct 30 01:58:43.455444 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:43.455412    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f185f9b-7950-4eed-8f05-c410e3cb57ba-lib-modules\") pod \"perf-node-gather-daemonset-k9w64\" (UID: \"0f185f9b-7950-4eed-8f05-c410e3cb57ba\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64"
Oct 30 01:58:43.455638 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:43.455464    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0f185f9b-7950-4eed-8f05-c410e3cb57ba-podres\") pod \"perf-node-gather-daemonset-k9w64\" (UID: \"0f185f9b-7950-4eed-8f05-c410e3cb57ba\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64"
Oct 30 01:58:43.455638 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:43.455502    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f185f9b-7950-4eed-8f05-c410e3cb57ba-sys\") pod \"perf-node-gather-daemonset-k9w64\" (UID: \"0f185f9b-7950-4eed-8f05-c410e3cb57ba\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64"
Oct 30 01:58:43.455638 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:43.455523    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7tfd\" (UniqueName: \"kubernetes.io/projected/0f185f9b-7950-4eed-8f05-c410e3cb57ba-kube-api-access-q7tfd\") pod \"perf-node-gather-daemonset-k9w64\" (UID: \"0f185f9b-7950-4eed-8f05-c410e3cb57ba\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64"
Oct 30 01:58:43.455638 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:43.455560    4864 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0f185f9b-7950-4eed-8f05-c410e3cb57ba-proc\") pod \"perf-node-gather-daemonset-k9w64\" (UID: \"0f185f9b-7950-4eed-8f05-c410e3cb57ba\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64"
Oct 30 01:58:43.455638 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:43.455570    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f185f9b-7950-4eed-8f05-c410e3cb57ba-lib-modules\") pod \"perf-node-gather-daemonset-k9w64\" (UID: \"0f185f9b-7950-4eed-8f05-c410e3cb57ba\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64"
Oct 30 01:58:43.455638 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:43.455612    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f185f9b-7950-4eed-8f05-c410e3cb57ba-sys\") pod \"perf-node-gather-daemonset-k9w64\" (UID: \"0f185f9b-7950-4eed-8f05-c410e3cb57ba\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64"
Oct 30 01:58:43.455913 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:43.455650    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0f185f9b-7950-4eed-8f05-c410e3cb57ba-proc\") pod \"perf-node-gather-daemonset-k9w64\" (UID: \"0f185f9b-7950-4eed-8f05-c410e3cb57ba\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64"
Oct 30 01:58:43.455913 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:43.455614    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0f185f9b-7950-4eed-8f05-c410e3cb57ba-podres\") pod \"perf-node-gather-daemonset-k9w64\" (UID: \"0f185f9b-7950-4eed-8f05-c410e3cb57ba\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64"
Oct 30 01:58:43.477464 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:43.477434    4864 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7tfd\" (UniqueName: \"kubernetes.io/projected/0f185f9b-7950-4eed-8f05-c410e3cb57ba-kube-api-access-q7tfd\") pod \"perf-node-gather-daemonset-k9w64\" (UID: \"0f185f9b-7950-4eed-8f05-c410e3cb57ba\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64"
Oct 30 01:58:43.564437 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:43.564350    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vgwmw/must-gather-wxzh7" event={"ID":"1b09bfdd-3763-48ec-91eb-3799ed2e3dfb","Type":"ContainerStarted","Data":"6ea9e55fa7ef0ad54b3d1d8deda0a5ca1ca0a25a4b678ed69977aa716e6ce571"}
Oct 30 01:58:43.566908 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:43.566877    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kqdbw/must-gather-sqq2l" event={"ID":"30ff5a8b-c19c-478d-9904-c5abdf4f260f","Type":"ContainerStarted","Data":"f819ec35a18b3a047e6764eb1b9ac980924e4d82d8265e89fb6908c57e946379"}
Oct 30 01:58:43.587860 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:43.587826    4864 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64"
Oct 30 01:58:43.653332 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:43.651129    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vgwmw/must-gather-wxzh7" podStartSLOduration=2.759409716 podStartE2EDuration="18.651108284s" totalImagesPullingTime="15.891698568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:43.608494868 +0000 UTC m=+4635.946596781" watchObservedRunningTime="2026-10-30 01:58:43.651108284 +0000 UTC m=+4635.989210191"
Oct 30 01:58:43.653332 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:43.651374    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-kqdbw/must-gather-sqq2l" podStartSLOduration=2.767196184 podStartE2EDuration="18.651366028s" totalImagesPullingTime="15.884169844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:43.648449254 +0000 UTC m=+4635.986551166" watchObservedRunningTime="2026-10-30 01:58:43.651366028 +0000 UTC m=+4635.989467933"
Oct 30 01:58:43.926452 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:43.926381    4864 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64"]
Oct 30 01:58:43.930239 ip-10-0-134-210 kubenswrapper[4864]: W1030 01:58:43.930199    4864 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0f185f9b_7950_4eed_8f05_c410e3cb57ba.slice/crio-e7b6fe243a86fb138f3920578c3beef72196b946fb76c013c177e0cd4ec39795 WatchSource:0}: Error finding container e7b6fe243a86fb138f3920578c3beef72196b946fb76c013c177e0cd4ec39795: Status 404 returned error can't find the container with id e7b6fe243a86fb138f3920578c3beef72196b946fb76c013c177e0cd4ec39795
Oct 30 01:58:44.044082 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:44.044032    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-lfnrl_2f315dfa-46fe-400e-aef8-8f5f2cbe2ccc/cluster-monitoring-operator/0.log"
Oct 30 01:58:44.581098 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:44.578900    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64" event={"ID":"0f185f9b-7950-4eed-8f05-c410e3cb57ba","Type":"ContainerStarted","Data":"e4c09e872fa73f34e4788b1bbcf491ffef54ba6b9db2796c3c5887a08957fdf2"}
Oct 30 01:58:44.581098 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:44.579004    4864 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64" event={"ID":"0f185f9b-7950-4eed-8f05-c410e3cb57ba","Type":"ContainerStarted","Data":"e7b6fe243a86fb138f3920578c3beef72196b946fb76c013c177e0cd4ec39795"}
Oct 30 01:58:44.581098 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:44.580287    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64"
Oct 30 01:58:44.617750 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:44.617682    4864 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64" podStartSLOduration=1.617662053 podStartE2EDuration="1.617662053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:44.614624431 +0000 UTC m=+4636.952726342" watchObservedRunningTime="2026-10-30 01:58:44.617662053 +0000 UTC m=+4636.955763964"
Oct 30 01:58:44.919430 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:44.919352    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-599679688f-rb5db_1cfebe80-0050-43c0-bff5-eecbfeb7ffb5/manager/0.log"
Oct 30 01:58:45.035021 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:45.034964    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-599679688f-rb5db_1cfebe80-0050-43c0-bff5-eecbfeb7ffb5/manager/0.log"
Oct 30 01:58:45.110987 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:45.110955    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nbrcn_145c8ccc-6041-44e5-8df0-c8c2420dfff2/manager/0.log"
Oct 30 01:58:45.217026 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:45.216992    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nbrcn_145c8ccc-6041-44e5-8df0-c8c2420dfff2/manager/0.log"
Oct 30 01:58:45.552462 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:45.552367    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mhjdw_8661055c-1d07-4776-96f7-ca8e36c1ea84/node-exporter/0.log"
Oct 30 01:58:45.682081 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:45.682037    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mhjdw_8661055c-1d07-4776-96f7-ca8e36c1ea84/kube-rbac-proxy/0.log"
Oct 30 01:58:45.811830 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:45.811744    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mhjdw_8661055c-1d07-4776-96f7-ca8e36c1ea84/init-textfile/0.log"
Oct 30 01:58:48.435108 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:48.435053    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-tlc9c_62dcd375-7d44-4774-b18f-fb601a98b643/proxy/0.log"
Oct 30 01:58:48.536687 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:48.536653    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-tlc9c_62dcd375-7d44-4774-b18f-fb601a98b643/proxy/0.log"
Oct 30 01:58:49.064126 ip-10-0-134-210 kubenswrapper[4864]: E1030 01:58:49.062949    4864 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f9vrm" containerName="extractor"
Oct 30 01:58:50.082959 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:50.082912    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-599679688f-rb5db_1cfebe80-0050-43c0-bff5-eecbfeb7ffb5/manager/0.log"
Oct 30 01:58:50.188508 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:50.188475    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-599679688f-rb5db_1cfebe80-0050-43c0-bff5-eecbfeb7ffb5/manager/0.log"
Oct 30 01:58:50.314500 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:50.314458    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nbrcn_145c8ccc-6041-44e5-8df0-c8c2420dfff2/manager/0.log"
Oct 30 01:58:50.387757 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:50.387669    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nbrcn_145c8ccc-6041-44e5-8df0-c8c2420dfff2/manager/0.log"
Oct 30 01:58:50.525232 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:50.525147    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-j7px8_6a145873-a56a-4fe8-bdb5-dd0e6837b948/networking-console-plugin/0.log"
Oct 30 01:58:51.608568 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:51.608521    4864 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-k9w64"
Oct 30 01:58:51.853971 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:51.853927    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/2.log"
Oct 30 01:58:51.860425 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:51.860352    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk65g_8e4dbba6-28af-46b3-ac90-37ce437fd318/console-operator/3.log"
Oct 30 01:58:53.634093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:53.631996    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-tlc9c_62dcd375-7d44-4774-b18f-fb601a98b643/proxy/0.log"
Oct 30 01:58:53.760818 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:53.759728    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-tlc9c_62dcd375-7d44-4774-b18f-fb601a98b643/proxy/0.log"
Oct 30 01:58:54.975906 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:54.975869    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-nkb8p_e05b9b53-5f5b-402e-8599-ce61481346e6/volume-data-source-validator/0.log"
Oct 30 01:58:55.622320 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:55.622283    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-599679688f-rb5db_1cfebe80-0050-43c0-bff5-eecbfeb7ffb5/manager/0.log"
Oct 30 01:58:55.715093 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:55.712625    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-599679688f-rb5db_1cfebe80-0050-43c0-bff5-eecbfeb7ffb5/manager/0.log"
Oct 30 01:58:55.849102 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:55.847381    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nbrcn_145c8ccc-6041-44e5-8df0-c8c2420dfff2/manager/0.log"
Oct 30 01:58:55.918628 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:55.918440    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nbrcn_145c8ccc-6041-44e5-8df0-c8c2420dfff2/manager/0.log"
Oct 30 01:58:58.661314 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:58.661279    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wq8t9_cee8ca5c-ba97-47b5-b22f-2e161bb93fdc/dns/0.log"
Oct 30 01:58:58.795971 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:58.795941    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wq8t9_cee8ca5c-ba97-47b5-b22f-2e161bb93fdc/kube-rbac-proxy/0.log"
Oct 30 01:58:58.923496 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:58.923422    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-49rdp_4770276e-e9dc-4f6c-94f6-d2dfe2f6f115/dns-node-resolver/0.log"
Oct 30 01:58:59.105793 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:58:59.105736    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-tlc9c_62dcd375-7d44-4774-b18f-fb601a98b643/proxy/0.log"
Oct 30 01:59:00.261158 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:59:00.261123    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-tlc9c_62dcd375-7d44-4774-b18f-fb601a98b643/proxy/0.log"
Oct 30 01:59:01.089425 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:59:01.089393    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qdjmq_19a807d5-c007-47b4-953c-28de606ababc/node-ca/0.log"
Oct 30 01:59:02.119114 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:59:02.118654    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57cd69bbcd-76gbt_6bd30c09-5438-468b-b353-eb0a1ba483b5/kube-auth-proxy/0.log"
Oct 30 01:59:02.547231 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:59:02.547197    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-w45z8_a13e1eb5-b23d-49ac-9e1d-6b431e9f07fd/payload-pre-processing/0.log"
Oct 30 01:59:02.618888 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:59:02.618813    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6k58k_d2a355c3-0703-422f-8ddb-555c4a364ade/payload-processing/0.log"
Oct 30 01:59:02.700416 ip-10-0-134-210 kubenswrapper[4864]: I1030 01:59:02.700377    4864 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-94c9fcc47-fdtjt_d95535e9-e677-463a-a662-7e8045c71d9b/router/0.log"